fix(tts): MeloTTS speaks up to 1.25 itself so fast speech sounds natural
Stretching all of the speed-up from 1.0 kept every sound but sounded awkward (UTMOS 3.13 at 1.5, against 3.73 for MeloTTS's own 1.5), while MeloTTS's own speed past 1.25 drops sounds (sentence-final "요" and first syllables, 4.6% character errors at 1.5). MeloTTS now speaks up to 1.25 and the WSOLA stretch does only the rest, with a 5 ms search: 2.6% and UTMOS 3.56 at 1.5 through the real child. Version 0.1.6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2
app/Cargo.lock
generated
2
app/Cargo.lock
generated
@@ -1484,7 +1484,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.5"
|
||||
version = "0.1.6"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.5"
|
||||
version = "0.1.6"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"
|
||||
|
||||
4
app/shell/Cargo.lock
generated
4
app/shell/Cargo.lock
generated
@@ -2380,7 +2380,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.5"
|
||||
version = "0.1.6"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
@@ -2408,7 +2408,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.5"
|
||||
version = "0.1.6"
|
||||
dependencies = [
|
||||
"jarvis-app",
|
||||
"serde_json",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.5"
|
||||
version = "0.1.6"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"
|
||||
|
||||
@@ -70,7 +70,7 @@ strings (numbers would lose precision).
|
||||
| `voice.wakeWords` | `["자비스"]` | Sent to the worker as STT `hotwords`, and the voice wake words of call detection. |
|
||||
| `voice.stt` | `{}` | Other worker STT settings (`partials`, `end_silence_ms`, `final_beam`, `language`, ...); the worker validates them. |
|
||||
| `chatChannels` | `[]` | `{id, name, description, read (true), send (true), edit (true), react (true), delete (false), pin (false), answer ("wakeWord"), syncDiscord (true)}` per channel; `answer` is in `conversation.spec.md`, `syncDiscord` (the channel's Discord name and topic follow `name` and `description`) in `admin.spec.md`. |
|
||||
| `voice.tts.speed` | `1.5` | How fast the voice speaks, 0.5-2.0 (1.0 is MeloTTS's natural pace; above it the worker stretches the speech and keeps its pitch, `voice_worker.spec.md`, "TTS"). Passed to the worker as `--melo-speed` when it speaks (`--melo-python`) and its arguments set no speed. A change saved while the services run goes to the worker at once (`tts_speed`) and applies from the next sentence, without a restart (saved while they start, it goes once they run, since the start may have read the settings before); it is sent again whenever the worker connects again. MeloTTS's Korean model has one voice, so there is no voice choice. |
|
||||
| `voice.tts.speed` | `1.5` | How fast the voice speaks, 0.5-2.0 (1.0 is MeloTTS's natural pace; MeloTTS speaks up to 1.25 itself and above that the worker stretches the speech and keeps its pitch, `voice_worker.spec.md`, "TTS"). Passed to the worker as `--melo-speed` when it speaks (`--melo-python`) and its arguments set no speed. A change saved while the services run goes to the worker at once (`tts_speed`) and applies from the next sentence, without a restart (saved while they start, it goes once they run, since the start may have read the settings before); it is sent again whenever the worker connects again. MeloTTS's Korean model has one voice, so there is no voice choice. |
|
||||
| `voiceWorker.python` | `python3` (`python.exe` on Windows) | Python of the worker venv. |
|
||||
| `voiceWorker.cwd` | `python-src` next to the app's executable when it is there (the Windows build), else the repo's `src/` | Folder holding the `voice_worker` package. |
|
||||
| `voiceWorker.args` | `[]` | Extra worker arguments, e.g. `["--melo-python", "<MeloTTS venv python>"]`. |
|
||||
|
||||
@@ -9,11 +9,13 @@ stdout: frames of u32 LE header length | header JSON | payload. The first frame
|
||||
int16 LE mono samples at the requested rate, or {"ok": false, "error": "..."} with no payload.
|
||||
Library output that would land on stdout is sent to stderr so it cannot corrupt the frames.
|
||||
|
||||
Speed: MeloTTS's own speed shortens every sound and slurs faster speech (Whisper large-v3 on five
|
||||
Korean sentences, three renders each: character errors 5.5% at 1.5 and 20% at 2.0, against 1.4% at
|
||||
1.0). So the model speaks at most at MODEL_SPEED_MAX (its natural pace) and a time stretch that keeps
|
||||
the pitch (WSOLA) makes it faster: 1.9% at 1.5 and 2.2% at 2.0. Letting the model go to 1.25 first
|
||||
measured slightly worse (2.2% and 3.1%).
|
||||
Speed: MeloTTS's own speed past about 1.25 drops sounds (five Korean sentences, three renders each,
|
||||
read back by Whisper large-v3: at 1.5 sentence-final "요" and first syllables went missing, 4.6%
|
||||
character errors, 9.6% at 1.75, against 1.2% at 1.0 and none added at 1.25). A time stretch that keeps
|
||||
the pitch (WSOLA) keeps every sound but sounds less natural the more it stretches (UTMOS 3.13 at 1.5
|
||||
stretched from 1.0, against 3.73 for the model's own 1.5). So the model speaks up to MODEL_SPEED_MAX
|
||||
and the stretch does only the rest: 2.6% and UTMOS 3.56 at 1.5, 1.9% and 3.09 at 1.75 (through this
|
||||
child). A 5 ms search scored higher than 10 ms (UTMOS 3.42 against 3.27 for a 1.2 stretch).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -26,7 +28,7 @@ import struct
|
||||
import sys
|
||||
import types
|
||||
|
||||
MODEL_SPEED_MAX = 1.0
|
||||
MODEL_SPEED_MAX = 1.25
|
||||
|
||||
|
||||
def split_speed(speed: float) -> tuple[float, float]:
|
||||
@@ -35,7 +37,7 @@ def split_speed(speed: float) -> tuple[float, float]:
|
||||
return model, speed / model
|
||||
|
||||
|
||||
def stretch(audio, factor: float, rate: int, frame_ms: float = 30.0, search_ms: float = 10.0):
|
||||
def stretch(audio, factor: float, rate: int, frame_ms: float = 30.0, search_ms: float = 5.0):
|
||||
"""Shorten ``audio`` by ``factor`` and keep its pitch (WSOLA): half-overlapping Hann frames are
|
||||
taken ``factor`` times further apart than they are laid down, each shifted within ``search_ms``
|
||||
to where it best continues the frame before it, so the pieces join without dips or clicks."""
|
||||
|
||||
@@ -181,11 +181,14 @@ synthesise is reported with `error` (`code: "tts_failed"`) and the request ends
|
||||
|
||||
Speed starts at `--melo-speed` and `tts_speed` changes it for every sentence made after it (a speed
|
||||
outside 0.5-2.0 or not a number is `bad_message` and the last good one stays). MeloTTS itself speaks
|
||||
at most at its natural pace (1.0): its own speed shortens every sound and slurs faster speech
|
||||
(Whisper large-v3 on five Korean sentences, three renders each: 5.5% character errors at 1.5 and 20%
|
||||
at 2.0, against 1.4% at 1.0). Faster speech is made at 1.0 and shortened with WSOLA, a time stretch
|
||||
that keeps the pitch (30 ms half-overlapping Hann frames, each shifted within 10 ms to where it best
|
||||
continues the one before): 1.9% at 1.5 and 2.2% at 2.0, about 15 ms per 4 s of speech.
|
||||
up to 1.25: its own speed shortens every sound, and past that it drops them (five Korean sentences,
|
||||
three renders each, read back by Whisper large-v3: at 1.5 sentence-final "요" and first syllables went
|
||||
missing, 4.6% character errors, 9.6% at 1.75, against 1.2% at 1.0). Faster speech is made at 1.25 and
|
||||
shortened with WSOLA, a time stretch that keeps the pitch (30 ms half-overlapping Hann frames, each
|
||||
shifted within 5 ms to where it best continues the one before): 2.6% at 1.5 and 1.9% at 1.75. The
|
||||
stretch costs naturalness the more it does (UTMOS at 1.5: 3.56, against 3.13 when all of it is
|
||||
stretched from 1.0), so it does only the part the model cannot; up to 1.25 nothing is stretched. About 15 ms per 4 s
|
||||
of speech.
|
||||
|
||||
### History and brain (M3)
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""How the MeloTTS child speeds speech up: the model never speaks faster than MODEL_SPEED_MAX (faster
|
||||
slurs), and the rest is a time stretch that keeps the pitch. And the engine sends its current speed
|
||||
drops sounds), and the rest is a time stretch that keeps the pitch. And the engine sends its current speed
|
||||
with every sentence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
Reference in New Issue
Block a user