feat(voice_worker): M1 multi-speaker streaming STT and sentence TTS over a local WebSocket
- One shared faster-whisper model for every speaker, batched; finals go before partials and partial batches are capped at two so a final never waits behind a ten-speaker pass. - Per-speaker 48k stereo to 16k mono conversion, streaming Silero VAD, utterance ends on VAD silence, Discord speaking_end, missing audio, or the 28 s Whisper window. - MeloTTS in its own venv through a child process; sentences stream out one chunk each, with incremental text, cancel and failure reporting. - Loopback-only server with a token from the environment; logs carry no transcript text. - Spec in src/voice_worker/voice_worker.spec.md; unit tests with fake engines and an opt-in real-GPU test that streams ten speakers at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -28,6 +28,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
|
||||
| `src/jarvis/memory/graph.spec.md` | Node graph memory (v2), self-organising tree, UI explorer | Dynamic structure; access-aware; auto-split/merge (future) |
|
||||
| `src/jarvis/memory/summariser.spec.md` | Diary summariser prompt contract, hygiene rules (deflection, attribution, topic separation), post-process scrub, and bulk-sweep clean button | Two-layer defence: prompt + deterministic scrub; corrupted summaries poison every downstream consumer |
|
||||
| `src/jarvis/memory/recall_gate.spec.md` | Deterministic skip-enrichment heuristic when the hot window covers a follow-up | Fail-open; language-agnostic via `\w{3,}` + `re.UNICODE`; planner intent always wins |
|
||||
| `src/voice_worker/voice_worker.spec.md` | v2 voice worker: multi-speaker streaming STT (shared Whisper, batched) and sentence TTS over a local WebSocket | Loopback only with an app token; no transcript text in logs; finals before partials; language-agnostic |
|
||||
|
||||
The LLM contexts graph at `docs/llm_contexts.md` maps every LLM call in the app (model, gating, inputs, outputs, limits, flow). Keep it up-to-date at all times: any change that adds, removes, or alters an LLM context (model resolution, timeout, cap, prompt source, gating flag, data-flow edge) must update `docs/llm_contexts.md` in the same PR.
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
| 단계 | 내용 | 완료 기준 |
|
||||
|---|---|---|
|
||||
| M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. 가상 모니터 확인을 뺀 모든 항목 측정 완료(2026-11-10). 가상 모니터 확인은 M6에서 합니다. 2026-11-10에 추가한 `desktop` 단계(agent-desktop)는 측정을 마쳤고, 자동 성능 조절용 `gamemode` 단계는 세 번 쟀고, `gamemode_stt`(LLM과 STT를 함께 켬, STT GPU·CPU)와 `gamemode_live`(실제 게임을 켬) 단계도 측정을 마쳤습니다(2026-11-10). 결과로 게임 중 최소 배치는 전문가 2GB로 정했고, 평소 배치에 STT 자리를 남기는 것과 게임 중 LLM이 답하는 동안 부분 전사 간격을 늘리는 것(게임이 없을 때는 그대로)도 정했습니다(사용자 결정, 2026-11-10). `gamemode_tune` 단계도 측정을 마쳤고(2026-11-11), 게임 중 LLM이 답하는 동안에는 부분 전사를 멈추기로 정했습니다. GPU 95% 제한은 측정에서 사용률을 낮추지 못해 기본값을 끔으로 바꿨고(사용자 결정, 2026-11-11), `video_drops` 단계(Jarvis를 돌리는 동안 유튜브 영상의 끊긴 프레임, 직접 실행)도 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 영상이 끊기지 않았고 제한이 끊김을 줄이지 못해 설정을 뺐습니다. 대신 게임 화면의 프레임을 직접 재는 `game_frames` 단계(게임을 켠 채로 Jarvis를 끈 때·쉴 때·바쁠 때와 우선순위를 낮춘 양보 상태, 직접 실행, 관리자 권한)를 넣었고 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 게임의 평균 fps와 1% low가 5% 넘게 떨어지지 않아 양보는 넣지 않기로 했습니다. 남은 M0 측정은 가상 모니터 확인(M6)입니다. 측정 키트: [scripts/m0/](../../scripts/m0/README.md) |
|
||||
| M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과 |
|
||||
| M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과. 구현과 테스트 완료(2026-11-11): `src/voice_worker/`, 스펙은 [voice_worker.spec.md](../../src/voice_worker/voice_worker.spec.md). 이 서버의 RTX 5050에서 10명이 겹쳐 말하는 실시간 입력(48kHz 스테레오, M0 한국어 샘플 8개를 10명에게 나눠 줌, 절반은 디스코드 speaking 종료, 절반은 무음으로 끝남)을 넣어 10명 모두 최종 전사를 받았고 문자 오류율은 평균 4%, 최대 19%("일곱 시"를 "7시"로 적은 것), 말 끝 판정부터 최종 전사 전송까지 중앙값 0.64초, 최대 1.56초였습니다. TTS는 첫 문장 소리가 0.2초에 나왔고, 그 소리를 다시 넣으면 같은 문장으로 받아 적혔습니다. 샘플이 MeloTTS 합성 음성이라 실제 사람 녹음과 게임PC 수치는 아직 없습니다 |
|
||||
| M2 | 디스코드 음성 코어 이전 | v1 봇 코드를 Electron 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기) |
|
||||
| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과 |
|
||||
| M4 | 왕복 완성과 지연 튜닝 | 호출어·호출 방식·지정한 사람 설정, 동시 대화 응답 정책 4가지(기본 "처음 대화만")와 정책별 끼어들기 규칙. 말 끝부터 첫 소리까지 실측 1.5초 이내, 10명 채널 부하 테스트 |
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
- **종료 판정 단축**: 고정 0.8초 침묵 대기 대신 디스코드 speaking 종료 이벤트와 VAD를 같이 써서 250~400ms 안에 확정합니다(설정에서 조절).
|
||||
- **2단 디코딩**: 부분 전사는 beam 1(빠름), 최종 전사는 beam 2~3(정확도)입니다.
|
||||
- **게임 중 LLM이 답하는 동안**: STT와 LLM은 같은 GPU를 씁니다. 10명이 쉬지 않고 말하며 부분 전사를 계속 돌리면 LLM 출력이 1/2.5~1/3로 줄었습니다(M0 `gamemode_stt`). 그래서 게임 중에 LLM이 답을 만드는 동안에는 부분 전사를 멈추고(그동안 실시간 자막이 바뀌지 않음), 말이 끝난 뒤의 최종 전사는 기다리지 않고 바로 처리합니다. 게임이 없을 때는 LLM이 답하는 동안에도 부분 전사를 그대로 합니다(사용자 결정, 2026-11-10). 최종 전사가 부분 전사보다 먼저입니다. 멈추기로 한 것은 M0 `gamemode_tune` 측정 결과입니다(2026-11-11): 2초 간격은 37.5 tok/s, 음성 첫 문장 0.89초에 최종 전사 대기가 최대 1.2초였고, 멈추면 64.9 tok/s, 0.55초였습니다. 부분 전사를 멈췄을 때 beam 2~3 최종 전사와 긴 말의 실제 지연은 아직 재지 않았습니다([pc-features.md](pc-features.md#자동-조절-측정-m0-gamemode_tune-2026-11-11)).
|
||||
- **부분 전사는 2명씩만 묶음(M1 실측)**: Whisper 인코더는 말 길이와 상관없이 늘 30초 창을 계산해서, 묶어도 비용이 사람 수에 비례합니다(RTX 5050에서 1명 0.2초, 10명 2.1초). 10명분 부분 전사를 한 번에 돌리면 그동안 들어온 최종 전사가 2초 넘게 기다려서, 부분 전사는 한 번에 2명까지만 돌리고 그 사이에 최종 전사가 먼저 들어가게 했습니다. 여러 명이 함께 말하면 자막은 늦어지고 사람마다 가장 최근 소리만 전사합니다. 인코더 창을 줄이면 4~7배 빨라졌지만 같은 문장을 되풀이하는 오류가 생겨 쓰지 않습니다.
|
||||
- **호출어 힌트**: 등록한 호출어를 STT에 힌트로 넘겨 잘못 알아듣는 일을 줄입니다.
|
||||
- **RealtimeSTT 사용 범위**: RealtimeSTT는 화자 하나에 인스턴스 하나 구조라 10명이면 모델을 10번 올립니다. 그래서 설계(부분/최종 전사, VAD 결합)만 참고하고, **모델 1개를 10명이 공유하는 배치 방식**으로 직접 구현합니다.
|
||||
- **선택지**: 더 낮은 지연이 필요하면 sherpa-onnx 스트리밍 모델을 "초고속" 옵션으로 검토합니다(한국어 정확도 확인 후).
|
||||
|
||||
4
src/voice_worker/__init__.py
Normal file
4
src/voice_worker/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
"""Jarvis v2 voice worker: multi-speaker streaming STT and sentence TTS over a local WebSocket.
|
||||
|
||||
See ``voice_worker.spec.md`` next to this file.
|
||||
"""
|
||||
66
src/voice_worker/__main__.py
Normal file
66
src/voice_worker/__main__.py
Normal file
@@ -0,0 +1,66 @@
|
||||
"""``JARVIS_VOICE_TOKEN=... python -m voice_worker`` (see voice_worker.spec.md)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(prog="voice_worker")
|
||||
ap.add_argument("--host", default="127.0.0.1")
|
||||
ap.add_argument("--port", type=int, default=8771)
|
||||
ap.add_argument("--stt-model", default="large-v3-turbo")
|
||||
ap.add_argument("--stt-device", default="cuda")
|
||||
ap.add_argument("--stt-compute", default="int8_float16")
|
||||
ap.add_argument("--melo-python", default=None, help="python of the MeloTTS venv; no TTS without it")
|
||||
ap.add_argument("--melo-device", default="cuda")
|
||||
ap.add_argument("--melo-speed", type=float, default=1.5)
|
||||
ap.add_argument("--melo-language", default="KR")
|
||||
args = ap.parse_args()
|
||||
|
||||
token = os.environ.get("JARVIS_VOICE_TOKEN", "")
|
||||
if not token:
|
||||
print("JARVIS_VOICE_TOKEN is not set", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
from .server import VoiceWorker, is_loopback
|
||||
from .stt_whisper import WhisperTranscriber
|
||||
from .vad import SileroVad
|
||||
|
||||
if not is_loopback(args.host):
|
||||
print("the voice worker only listens on a loopback address", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
transcriber = WhisperTranscriber(args.stt_model, args.stt_device, args.stt_compute)
|
||||
transcriber.warm_up()
|
||||
vad = SileroVad()
|
||||
tts = None
|
||||
if args.melo_python:
|
||||
from .tts_melo import MeloEngine
|
||||
tts = MeloEngine(args.melo_python, args.melo_device, args.melo_speed, args.melo_language)
|
||||
|
||||
async def run() -> None:
|
||||
worker = VoiceWorker(token, transcriber, vad.stream, tts)
|
||||
server = await worker.serve(args.host, args.port)
|
||||
debug_log(f"voice worker listening on {args.host}:{args.port} tts={'on' if tts else 'off'}",
|
||||
"voice_worker")
|
||||
print(f"READY {args.host}:{args.port}", flush=True)
|
||||
await server.serve_forever()
|
||||
|
||||
try:
|
||||
asyncio.run(run())
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
if tts:
|
||||
tts.close()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
58
src/voice_worker/audio.py
Normal file
58
src/voice_worker/audio.py
Normal file
@@ -0,0 +1,58 @@
|
||||
"""Incoming PCM to 16 kHz mono float32, and clip normalisation before transcription."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
|
||||
TARGET_RATE = 16000
|
||||
_TAPS = 97 # windowed-sinc low-pass for the 48 kHz -> 16 kHz decimation
|
||||
_CUTOFF_HZ = 7200 # below the new 8 kHz Nyquist
|
||||
|
||||
|
||||
def _lowpass(rate: int) -> np.ndarray:
|
||||
n = np.arange(_TAPS) - (_TAPS - 1) / 2
|
||||
fc = _CUTOFF_HZ / rate
|
||||
h = 2 * fc * np.sinc(2 * fc * n) * np.blackman(_TAPS)
|
||||
return (h / h.sum()).astype(np.float64)
|
||||
|
||||
|
||||
class InputConverter:
|
||||
"""Streaming converter for one speaker. Keeps filter history and odd bytes between frames."""
|
||||
|
||||
def __init__(self, rate: int, channels: int):
|
||||
if rate not in (16000, 48000):
|
||||
raise ValueError("audio rate must be 16000 or 48000")
|
||||
if channels not in (1, 2):
|
||||
raise ValueError("audio channels must be 1 or 2")
|
||||
self.channels = channels
|
||||
self.factor = rate // TARGET_RATE
|
||||
self._carry = b""
|
||||
if self.factor > 1:
|
||||
self._h = _lowpass(rate)
|
||||
self._hist = np.zeros(_TAPS - 1, dtype=np.float64)
|
||||
self._phase = 0 # input samples to skip before the next kept output
|
||||
|
||||
def push(self, pcm: bytes) -> np.ndarray:
|
||||
data = self._carry + pcm
|
||||
frame_bytes = 2 * self.channels
|
||||
usable = len(data) - len(data) % frame_bytes
|
||||
self._carry = data[usable:]
|
||||
x = np.frombuffer(data[:usable], dtype="<i2").astype(np.float64) / 32768.0
|
||||
if self.channels == 2:
|
||||
x = x.reshape(-1, 2).mean(axis=1)
|
||||
if self.factor == 1:
|
||||
return x.astype(np.float32)
|
||||
buf = np.concatenate([self._hist, x])
|
||||
self._hist = buf[-(_TAPS - 1):]
|
||||
y = np.convolve(buf, self._h, mode="valid") # one output per new input sample
|
||||
out = y[self._phase::self.factor]
|
||||
self._phase = (self._phase - len(y)) % self.factor
|
||||
return out.astype(np.float32)
|
||||
|
||||
|
||||
def peak_normalise(x: np.ndarray, target: float = 0.9, max_gain: float = 10.0) -> np.ndarray:
|
||||
peak = float(np.max(np.abs(x))) if len(x) else 0.0
|
||||
if peak <= 0:
|
||||
return x.astype(np.float32)
|
||||
gain = min(target / peak, max_gain)
|
||||
return (x * gain).astype(np.float32)
|
||||
103
src/voice_worker/melo_child.py
Normal file
103
src/voice_worker/melo_child.py
Normal file
@@ -0,0 +1,103 @@
|
||||
"""MeloTTS child process for the voice worker. Runs in its own Python 3.11 venv (transformers 4.27).
|
||||
|
||||
Stand-alone on purpose: it must not import the worker package, which lives in another venv.
|
||||
|
||||
stdin: one JSON request per line: {"text": "...", "rate": 48000}
|
||||
stdout: frames of u32 LE header length | header JSON | payload. The first frame is
|
||||
{"ready": true, "rate": <native rate>}; each reply is {"ok": true, "samples": N} followed by N
|
||||
int16 LE mono samples at the requested rate, or {"ok": false, "error": "..."} with no payload.
|
||||
Library output that would land on stdout is sent to stderr so it cannot corrupt the frames.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import types
|
||||
|
||||
|
||||
def korean_only_patches() -> None:
|
||||
"""MeloTTS imports its Japanese front end (module-level MeCab.Tagger()) even for Korean, and on
|
||||
Windows g2pkk asks for ``eunjeon`` instead of python-mecab-ko. Same fix as the M0 kit."""
|
||||
try:
|
||||
import MeCab # noqa: F401
|
||||
MeCab.Tagger # noqa: B018
|
||||
except Exception:
|
||||
stub = types.ModuleType("MeCab")
|
||||
|
||||
class Tagger:
|
||||
def __init__(self, *a, **k):
|
||||
pass
|
||||
|
||||
def parse(self, text):
|
||||
raise RuntimeError("Japanese MeCab is not installed")
|
||||
|
||||
stub.Tagger = Tagger
|
||||
sys.modules["MeCab"] = stub
|
||||
try:
|
||||
import g2pkk.g2pkk as g2p_mod
|
||||
except Exception:
|
||||
return
|
||||
if g2p_mod.platform.system() == "Windows":
|
||||
g2p_mod.G2p.check_mecab = lambda self: None
|
||||
g2p_mod.G2p.get_mecab = lambda self: importlib.import_module("mecab").MeCab()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
# Python puts this file's folder first on sys.path; drop it so the worker's own modules there
|
||||
# never shadow the MeloTTS packages.
|
||||
here = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path[:] = [p for p in sys.path if os.path.abspath(p or ".") != here]
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--language", default="KR")
|
||||
ap.add_argument("--device", default="cuda")
|
||||
ap.add_argument("--speed", type=float, default=1.5)
|
||||
args = ap.parse_args()
|
||||
|
||||
out = os.fdopen(os.dup(sys.stdout.fileno()), "wb")
|
||||
os.dup2(sys.stderr.fileno(), sys.stdout.fileno())
|
||||
sys.stdout = sys.stderr
|
||||
|
||||
def reply(header: dict, payload: bytes = b"") -> None:
|
||||
raw = json.dumps(header).encode()
|
||||
out.write(struct.pack("<I", len(raw)) + raw + payload)
|
||||
out.flush()
|
||||
|
||||
try:
|
||||
korean_only_patches()
|
||||
import numpy as np
|
||||
from melo.api import TTS
|
||||
from scipy.signal import resample_poly
|
||||
model = TTS(language=args.language, device=args.device)
|
||||
spk_map = model.hps.data.spk2id
|
||||
speaker = spk_map[args.language] if args.language in spk_map else spk_map[list(spk_map.keys())[0]]
|
||||
native = int(model.hps.data.sampling_rate)
|
||||
model.tts_to_file("워밍업입니다.", speaker, None, speed=args.speed, quiet=True)
|
||||
except Exception as e:
|
||||
reply({"ready": False, "error": f"{type(e).__name__}: {e}"})
|
||||
return 1
|
||||
reply({"ready": True, "rate": native})
|
||||
|
||||
for line in sys.stdin:
|
||||
try:
|
||||
req = json.loads(line)
|
||||
rate = int(req["rate"])
|
||||
audio = model.tts_to_file(str(req["text"]), speaker, None, speed=args.speed, quiet=True)
|
||||
audio = np.asarray(audio, dtype=np.float64)
|
||||
if rate != native:
|
||||
from math import gcd
|
||||
g = gcd(rate, native)
|
||||
audio = resample_poly(audio, rate // g, native // g)
|
||||
pcm = (np.clip(audio, -1.0, 1.0) * 32767).astype("<i2")
|
||||
reply({"ok": True, "samples": int(len(pcm))}, pcm.tobytes())
|
||||
except Exception as e:
|
||||
reply({"ok": False, "error": f"{type(e).__name__}: {e}"})
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
56
src/voice_worker/protocol.py
Normal file
56
src/voice_worker/protocol.py
Normal file
@@ -0,0 +1,56 @@
|
||||
"""Binary frame layout shared with the Electron voice core (see voice_worker.spec.md)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import struct
|
||||
|
||||
KIND_AUDIO = 0x01
|
||||
KIND_TTS = 0x02
|
||||
|
||||
|
||||
class ProtocolError(ValueError):
|
||||
"""A frame or message that does not follow the protocol."""
|
||||
|
||||
|
||||
def _encode_id(value: str) -> bytes:
|
||||
raw = value.encode("utf-8")
|
||||
if not 0 < len(raw) < 256:
|
||||
raise ProtocolError("id must be 1-255 bytes of UTF-8")
|
||||
return bytes([len(raw)]) + raw
|
||||
|
||||
|
||||
def _decode_id(frame: bytes, kind: int) -> tuple[str, int]:
|
||||
if len(frame) < 2 or frame[0] != kind:
|
||||
raise ProtocolError("unexpected frame kind")
|
||||
n = frame[1]
|
||||
end = 2 + n
|
||||
if n == 0 or len(frame) < end:
|
||||
raise ProtocolError("truncated id")
|
||||
try:
|
||||
return frame[2:end].decode("utf-8"), end
|
||||
except UnicodeDecodeError as e:
|
||||
raise ProtocolError("id is not UTF-8") from e
|
||||
|
||||
|
||||
def encode_audio_frame(speaker: str, pcm: bytes) -> bytes:
|
||||
return bytes([KIND_AUDIO]) + _encode_id(speaker) + pcm
|
||||
|
||||
|
||||
def decode_audio_frame(frame: bytes) -> tuple[str, bytes]:
|
||||
speaker, end = _decode_id(frame, KIND_AUDIO)
|
||||
pcm = frame[end:]
|
||||
if not pcm or len(pcm) % 2:
|
||||
raise ProtocolError("audio payload must be a non-empty run of 16-bit samples")
|
||||
return speaker, pcm
|
||||
|
||||
|
||||
def encode_tts_frame(request_id: str, seq: int, pcm: bytes) -> bytes:
|
||||
return bytes([KIND_TTS]) + _encode_id(request_id) + struct.pack("<I", seq) + pcm
|
||||
|
||||
|
||||
def decode_tts_frame(frame: bytes) -> tuple[str, int, bytes]:
|
||||
request_id, end = _decode_id(frame, KIND_TTS)
|
||||
if len(frame) < end + 4:
|
||||
raise ProtocolError("truncated sequence number")
|
||||
(seq,) = struct.unpack_from("<I", frame, end)
|
||||
return request_id, seq, frame[end + 4:]
|
||||
12
src/voice_worker/requirements.txt
Normal file
12
src/voice_worker/requirements.txt
Normal file
@@ -0,0 +1,12 @@
|
||||
# Voice worker venv (Python 3.12). Same STT pins the M0 kit measured on the game PC.
|
||||
faster-whisper==1.2.1
|
||||
ctranslate2==4.8.1
|
||||
nvidia-cublas-cu12==12.9.2.10
|
||||
nvidia-cudnn-cu12==9.24.0.43
|
||||
onnxruntime>=1.17
|
||||
websockets>=13
|
||||
numpy
|
||||
# jarvis.debug (debug_log) reads the jarvis config, which needs python-dotenv.
|
||||
python-dotenv==1.0.1
|
||||
# MeloTTS runs in its own Python 3.11 venv (see the "melo" entry in scripts/m0/m0lib/envs.py) and is
|
||||
# reached through melo_child.py; pass that venv's python with --melo-python.
|
||||
128
src/voice_worker/scheduler.py
Normal file
128
src/voice_worker/scheduler.py
Normal file
@@ -0,0 +1,128 @@
|
||||
"""One shared STT model for every speaker: finals first, then partials, each batched.
|
||||
|
||||
All queue state lives on the asyncio loop; the model runs on one executor thread.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable, Hashable, Protocol
|
||||
|
||||
import numpy as np
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DecodeOptions:
|
||||
beam: int
|
||||
language: str | None
|
||||
hotwords: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass
|
||||
class SttResult:
|
||||
text: str
|
||||
stt_ms: int
|
||||
error: str | None = None
|
||||
|
||||
|
||||
class Transcriber(Protocol):
|
||||
def transcribe(self, clips: list[np.ndarray], opts: DecodeOptions) -> list[str]: ...
|
||||
|
||||
|
||||
Callback = Callable[[SttResult], None]
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Job:
|
||||
key: Hashable
|
||||
audio: np.ndarray
|
||||
opts: DecodeOptions
|
||||
callback: Callback
|
||||
|
||||
|
||||
class SttScheduler:
|
||||
def __init__(self, transcriber: Transcriber, max_batch: int = 16, max_partial_batch: int = 2):
|
||||
self.transcriber = transcriber
|
||||
self.max_batch = max_batch
|
||||
# Whisper's encoder always runs on a 30 s window, so its cost grows with every clip in a batch
|
||||
# and batching saves little. A small partial batch bounds how long a final can wait behind one.
|
||||
self.max_partial_batch = max_partial_batch
|
||||
self._finals: list[_Job] = []
|
||||
self._partials: "OrderedDict[Hashable, _Job]" = OrderedDict()
|
||||
self._wake = asyncio.Event()
|
||||
self._executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="stt")
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
self._task = asyncio.get_running_loop().create_task(self._run())
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._executor.shutdown(wait=False, cancel_futures=True)
|
||||
|
||||
def submit_final(self, key: Hashable, audio: np.ndarray, opts: DecodeOptions, callback: Callback) -> None:
|
||||
self._finals.append(_Job(key, audio, opts, callback))
|
||||
self._wake.set()
|
||||
|
||||
def submit_partial(self, key: Hashable, audio: np.ndarray, opts: DecodeOptions, callback: Callback) -> None:
|
||||
self._partials.pop(key, None) # a newer partial replaces the waiting one
|
||||
self._partials[key] = _Job(key, audio, opts, callback)
|
||||
self._wake.set()
|
||||
|
||||
def drop_partials(self, predicate: Callable[[Hashable], bool]) -> None:
|
||||
for key in [k for k in self._partials if predicate(k)]:
|
||||
del self._partials[key]
|
||||
|
||||
def _take(self) -> tuple[str, list[_Job]]:
|
||||
if self._finals:
|
||||
kind, pool = "final", self._finals
|
||||
elif self._partials:
|
||||
kind, pool = "partial", list(self._partials.values())
|
||||
else:
|
||||
return "", []
|
||||
opts = pool[0].opts
|
||||
cap = self.max_batch if kind == "final" else self.max_partial_batch
|
||||
batch = [j for j in pool if j.opts == opts][:cap]
|
||||
if kind == "final":
|
||||
taken = {id(j) for j in batch}
|
||||
self._finals = [j for j in self._finals if id(j) not in taken]
|
||||
else:
|
||||
for j in batch:
|
||||
del self._partials[j.key]
|
||||
return kind, batch
|
||||
|
||||
async def _run(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
while True:
|
||||
kind, batch = self._take()
|
||||
if not batch:
|
||||
self._wake.clear()
|
||||
await self._wake.wait()
|
||||
continue
|
||||
clips = [j.audio for j in batch]
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
texts = await loop.run_in_executor(self._executor, self.transcriber.transcribe, clips,
|
||||
batch[0].opts)
|
||||
error = None
|
||||
except Exception as e: # the model failing must not stop every speaker's STT
|
||||
texts, error = [""] * len(batch), f"{type(e).__name__}: {e}"
|
||||
ms = int((time.perf_counter() - t0) * 1000)
|
||||
debug_log(f"stt batch kind={kind} size={len(batch)} beam={batch[0].opts.beam} ms={ms}"
|
||||
+ (f" error={error}" if error else ""), "voice_worker")
|
||||
for job, text in zip(batch, texts):
|
||||
try:
|
||||
job.callback(SttResult(text, ms, error))
|
||||
except Exception as e: # pragma: no cover - a broken session must not stop the loop
|
||||
debug_log(f"stt callback failed: {type(e).__name__}: {e}", "voice_worker")
|
||||
138
src/voice_worker/segmenter.py
Normal file
138
src/voice_worker/segmenter.py
Normal file
@@ -0,0 +1,138 @@
|
||||
"""Per-speaker utterance segmentation on 16 kHz mono audio (rules in voice_worker.spec.md)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .settings import SttSettings
|
||||
|
||||
SR = 16000
|
||||
WINDOW = 512 # Silero window at 16 kHz
|
||||
WINDOW_MS = WINDOW * 1000 / SR # 32 ms
|
||||
START_PROB = 0.5
|
||||
KEEP_PROB = 0.35
|
||||
PRE_ROLL_MS = 200
|
||||
MIN_SPEECH_MS = 250
|
||||
TAIL_KEEP_MS = 200
|
||||
MAX_UTTERANCE_MS = 28000
|
||||
|
||||
|
||||
class VadStream(Protocol):
|
||||
def prob(self, window: np.ndarray) -> float: ...
|
||||
|
||||
|
||||
@dataclass
|
||||
class SpeechStart:
|
||||
utt: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class Partial:
|
||||
utt: int
|
||||
audio: np.ndarray
|
||||
|
||||
|
||||
@dataclass
|
||||
class Final:
|
||||
utt: int
|
||||
audio: np.ndarray
|
||||
reason: str # silence | speaking_end | idle | max_length
|
||||
decided_at: float
|
||||
|
||||
|
||||
def _windows(ms: float) -> int:
|
||||
return int(ms // WINDOW_MS)
|
||||
|
||||
|
||||
class Segmenter:
|
||||
def __init__(self, vad: VadStream, settings: SttSettings):
|
||||
self.vad = vad
|
||||
self.settings = settings
|
||||
self._pending = np.zeros(0, dtype=np.float32)
|
||||
self._pre: list[np.ndarray] = []
|
||||
self._utt_count = 0
|
||||
self._last_audio_at: float | None = None
|
||||
self._reset()
|
||||
|
||||
def _reset(self) -> None:
|
||||
self._active = False
|
||||
self._win: list[np.ndarray] = []
|
||||
self._speech = 0
|
||||
self._trailing = 0
|
||||
self._started = False
|
||||
self._last_partial_at = 0.0
|
||||
self._last_partial_len = 0
|
||||
|
||||
@property
|
||||
def open_utt(self) -> int | None:
|
||||
"""The utterance that sent speech_start and has not ended yet."""
|
||||
return self._utt_count if self._active and self._started else None
|
||||
|
||||
def push(self, samples: np.ndarray, now: float) -> list:
|
||||
self._last_audio_at = now
|
||||
data = np.concatenate([self._pending, samples.astype(np.float32, copy=False)])
|
||||
n = len(data) // WINDOW * WINDOW
|
||||
self._pending = data[n:]
|
||||
events: list = []
|
||||
for i in range(0, n, WINDOW):
|
||||
events += self._window(data[i:i + WINDOW], now)
|
||||
if self.open_utt is not None and len(self._win) > self._last_partial_len \
|
||||
and now - self._last_partial_at >= self.settings.partial_interval_ms / 1000:
|
||||
self._last_partial_at = now
|
||||
self._last_partial_len = len(self._win)
|
||||
events.append(Partial(self._utt_count, np.concatenate(self._win)))
|
||||
return events
|
||||
|
||||
def _window(self, w: np.ndarray, now: float) -> list:
|
||||
p = self.vad.prob(w)
|
||||
if not self._active:
|
||||
if p >= START_PROB:
|
||||
self._active = True
|
||||
self._win = self._pre + [w]
|
||||
self._pre = []
|
||||
self._speech = 1
|
||||
self._trailing = 0
|
||||
else:
|
||||
self._pre.append(w)
|
||||
del self._pre[:-_windows(PRE_ROLL_MS)]
|
||||
return []
|
||||
self._win.append(w)
|
||||
if p >= KEEP_PROB:
|
||||
self._speech += 1
|
||||
self._trailing = 0
|
||||
else:
|
||||
self._trailing += 1
|
||||
events: list = []
|
||||
if not self._started and self._speech * WINDOW_MS >= MIN_SPEECH_MS:
|
||||
self._started = True
|
||||
self._utt_count += 1
|
||||
self._last_partial_at = now
|
||||
events.append(SpeechStart(self._utt_count))
|
||||
if self._trailing * WINDOW_MS >= self.settings.end_silence_ms:
|
||||
events += self._end("silence", now)
|
||||
elif len(self._win) * WINDOW_MS >= MAX_UTTERANCE_MS:
|
||||
events += self._end("max_length", now)
|
||||
return events
|
||||
|
||||
def _end(self, reason: str, now: float) -> list:
|
||||
events: list = []
|
||||
if self._active and self._started:
|
||||
drop = max(0, self._trailing - _windows(TAIL_KEEP_MS))
|
||||
kept = self._win[:len(self._win) - drop]
|
||||
events.append(Final(self._utt_count, np.concatenate(kept), reason, now))
|
||||
self._reset()
|
||||
return events
|
||||
|
||||
def speaking_end(self, now: float) -> list:
|
||||
self._pending = np.zeros(0, dtype=np.float32)
|
||||
return self._end("speaking_end", now) if self._active else []
|
||||
|
||||
def tick(self, now: float) -> list:
|
||||
if self._active and self._last_audio_at is not None \
|
||||
and now - self._last_audio_at >= self.settings.end_silence_ms / 1000:
|
||||
self._pending = np.zeros(0, dtype=np.float32)
|
||||
return self._end("idle", now)
|
||||
return []
|
||||
76
src/voice_worker/sentences.py
Normal file
76
src/voice_worker/sentences.py
Normal file
@@ -0,0 +1,76 @@
|
||||
"""Punctuation-only sentence splitting for streaming TTS (language-agnostic).
|
||||
|
||||
Same rule as the v1 bridge (``bridge/text_utils.py``), which is legacy and goes away with v2; the
|
||||
worker keeps its own copy so it does not depend on the v1 package.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# A boundary is a run of newlines, a run of CJK terminators (always a sentence end, CJK puts no space
|
||||
# after one), or a run of ASCII terminators that really ends a sentence: followed by whitespace, a
|
||||
# closing quote or bracket, or the end of the text. Dots inside tokens (17.5, v2.0, example.com) are
|
||||
# followed by a non-space character, so they never match.
|
||||
_BOUNDARY = re.compile(
|
||||
r"""
|
||||
(?P<nl>\n+)
|
||||
| (?P<cjk>[。!?…]+)
|
||||
| (?P<ascii>[.!?]+)(?=[)\]"'”’》」』]*(?:\s|$))
|
||||
""",
|
||||
re.VERBOSE,
|
||||
)
|
||||
|
||||
|
||||
def split_sentences(text: str | None, min_len: int = 5) -> list[str]:
|
||||
"""Split ``text`` into sentence chunks; fragments under ``min_len`` characters join a neighbour."""
|
||||
return _split(text, min_len, merge_short_tail=True)
|
||||
|
||||
|
||||
def _split(text: str | None, min_len: int, merge_short_tail: bool) -> list[str]:
|
||||
text = (text or "").strip()
|
||||
if not text:
|
||||
return []
|
||||
chunks: list[str] = []
|
||||
buf = ""
|
||||
last = 0
|
||||
for m in _BOUNDARY.finditer(text):
|
||||
buf += text[last:m.end()]
|
||||
last = m.end()
|
||||
if len(buf.strip()) >= min_len:
|
||||
chunks.append(buf.strip())
|
||||
buf = ""
|
||||
tail = (buf + text[last:]).strip()
|
||||
if tail:
|
||||
if merge_short_tail and chunks and len(tail) < min_len:
|
||||
chunks[-1] = chunks[-1] + " " + tail
|
||||
else:
|
||||
chunks.append(tail)
|
||||
return chunks
|
||||
|
||||
|
||||
class StreamingSplitter:
|
||||
"""Feed text as it streams in; get back every sentence that is certainly complete."""
|
||||
|
||||
def __init__(self, min_len: int = 5):
|
||||
self.min_len = min_len
|
||||
self._buf = ""
|
||||
|
||||
def feed(self, text: str) -> list[str]:
|
||||
self._buf += text
|
||||
# The tail may still grow, so it is not merged backwards here.
|
||||
chunks = _split(self._buf, self.min_len, merge_short_tail=False)
|
||||
if len(chunks) <= 1:
|
||||
return []
|
||||
done = chunks[:-1]
|
||||
# Keep the raw text after the last emitted chunk, so spacing inside the tail is untouched.
|
||||
consumed = 0
|
||||
for c in done:
|
||||
consumed = self._buf.index(c, consumed) + len(c)
|
||||
self._buf = self._buf[consumed:]
|
||||
return done
|
||||
|
||||
def finish(self) -> list[str]:
|
||||
rest = self._buf.strip()
|
||||
self._buf = ""
|
||||
return [rest] if rest else []
|
||||
310
src/voice_worker/server.py
Normal file
310
src/voice_worker/server.py
Normal file
@@ -0,0 +1,310 @@
|
||||
"""Local WebSocket server: one session per connection (protocol in voice_worker.spec.md)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hmac
|
||||
import ipaddress
|
||||
import itertools
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable
|
||||
|
||||
import numpy as np
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
from .audio import InputConverter
|
||||
from .protocol import KIND_AUDIO, ProtocolError, decode_audio_frame, encode_tts_frame
|
||||
from .scheduler import DecodeOptions, SttResult, SttScheduler, Transcriber
|
||||
from .segmenter import SR, Final, Partial, Segmenter, SpeechStart, VadStream
|
||||
from .settings import SttSettings
|
||||
from .tts import TtsEngine, TtsRequest, TtsService
|
||||
|
||||
CLOSE_BAD_HELLO = 4400
|
||||
CLOSE_BAD_TOKEN = 4401
|
||||
MAX_SPEAKERS = 32
|
||||
|
||||
|
||||
def is_loopback(host: str) -> bool:
|
||||
if host == "localhost":
|
||||
return True
|
||||
try:
|
||||
return ipaddress.ip_address(host).is_loopback
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
class VoiceWorker:
|
||||
def __init__(self, token: str, transcriber: Transcriber | None, vad_factory: Callable[[], VadStream],
|
||||
tts_engine: TtsEngine | None = None, hello_timeout_s: float = 5.0, tick_s: float = 0.05):
|
||||
if not token:
|
||||
raise ValueError("a connection token is required")
|
||||
self.token = token
|
||||
self.transcriber = transcriber
|
||||
self.vad_factory = vad_factory
|
||||
self.tts_engine = tts_engine
|
||||
self.hello_timeout_s = hello_timeout_s
|
||||
self.tick_s = tick_s
|
||||
self.scheduler: SttScheduler | None = None
|
||||
self.tts: TtsService | None = None
|
||||
self._ids = itertools.count(1)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self.transcriber is not None:
|
||||
self.scheduler = SttScheduler(self.transcriber)
|
||||
self.scheduler.start()
|
||||
if self.tts_engine is not None:
|
||||
self.tts = TtsService(self.tts_engine)
|
||||
self.tts.start()
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self.scheduler:
|
||||
await self.scheduler.stop()
|
||||
if self.tts:
|
||||
await self.tts.stop()
|
||||
|
||||
async def serve(self, host: str, port: int):
|
||||
"""Start listening; returns the websockets server. Loopback addresses only."""
|
||||
if not is_loopback(host):
|
||||
raise ValueError("the voice worker only listens on a loopback address")
|
||||
from websockets.asyncio.server import serve
|
||||
await self.start()
|
||||
return await serve(self.handle, host, port, max_size=2 ** 20)
|
||||
|
||||
async def handle(self, ws) -> None:
|
||||
try:
|
||||
first = await asyncio.wait_for(ws.recv(), self.hello_timeout_s)
|
||||
hello = json.loads(first) if isinstance(first, str) else None
|
||||
except (asyncio.TimeoutError, ValueError):
|
||||
hello = None
|
||||
except Exception:
|
||||
return
|
||||
if not isinstance(hello, dict) or hello.get("type") != "hello":
|
||||
debug_log("connection refused: no hello", "voice_worker")
|
||||
await ws.close(CLOSE_BAD_HELLO, "hello expected")
|
||||
return
|
||||
token = hello.get("token")
|
||||
if not isinstance(token, str) or not hmac.compare_digest(token.encode(), self.token.encode()):
|
||||
debug_log("connection refused: bad token", "voice_worker")
|
||||
await ws.close(CLOSE_BAD_TOKEN, "bad token")
|
||||
return
|
||||
try:
|
||||
session = Session(self, ws, next(self._ids), hello)
|
||||
except ValueError as e:
|
||||
await ws.close(CLOSE_BAD_HELLO, str(e)[:100])
|
||||
return
|
||||
await session.run()
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Speaker:
|
||||
converter: InputConverter
|
||||
segmenter: Segmenter
|
||||
|
||||
|
||||
def _format(value, default: dict, rates: tuple[int, ...] | None) -> dict:
|
||||
if value is None:
|
||||
return dict(default)
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("format must be an object")
|
||||
rate, channels = value.get("rate", default["rate"]), value.get("channels", default["channels"])
|
||||
ok_rate = isinstance(rate, int) and (rate in rates if rates else 8000 <= rate <= 96000)
|
||||
if not ok_rate or channels not in (1, 2) or isinstance(channels, bool):
|
||||
raise ValueError("unsupported audio format")
|
||||
return {"rate": rate, "channels": channels}
|
||||
|
||||
|
||||
class Session:
|
||||
def __init__(self, worker: VoiceWorker, ws, sid: int, hello: dict):
|
||||
self.worker = worker
|
||||
self.ws = ws
|
||||
self.sid = sid
|
||||
self.audio = _format(hello.get("audio"), {"rate": 48000, "channels": 2}, (16000, 48000))
|
||||
self.tts_format = _format(hello.get("tts"), {"rate": 48000, "channels": 2}, None)
|
||||
self.settings = SttSettings()
|
||||
stt = hello.get("stt")
|
||||
if stt is not None:
|
||||
if not isinstance(stt, dict):
|
||||
raise ValueError("stt must be an object")
|
||||
self.settings.update(stt)
|
||||
self.speakers: dict[str, _Speaker] = {}
|
||||
self.tts_requests: dict[str, TtsRequest] = {}
|
||||
self.out: asyncio.Queue = asyncio.Queue()
|
||||
self.closed = False
|
||||
|
||||
# --- outbound -------------------------------------------------------------------------------
|
||||
def send(self, msg) -> None:
|
||||
if not self.closed:
|
||||
self.out.put_nowait(msg if isinstance(msg, bytes) else json.dumps(msg, ensure_ascii=False))
|
||||
|
||||
def error(self, code: str, message: str, **extra) -> None:
|
||||
self.send({"type": "error", "code": code, "message": message, **extra})
|
||||
|
||||
async def _writer(self) -> None:
|
||||
while True:
|
||||
await self.ws.send(await self.out.get())
|
||||
|
||||
# --- lifecycle ------------------------------------------------------------------------------
|
||||
async def run(self) -> None:
|
||||
w = self.worker
|
||||
debug_log(f"session {self.sid} connected audio={self.audio} tts={self.tts_format}", "voice_worker")
|
||||
self.send({"type": "hello_ok", "stt_ready": w.scheduler is not None, "tts_ready": w.tts is not None,
|
||||
"audio": self.audio, "tts": self.tts_format, "stt": self.settings.as_dict()})
|
||||
writer = asyncio.create_task(self._writer())
|
||||
ticker = asyncio.create_task(self._ticker())
|
||||
try:
|
||||
async for frame in self.ws:
|
||||
if isinstance(frame, bytes):
|
||||
self._on_binary(frame)
|
||||
else:
|
||||
self._on_text(frame)
|
||||
except Exception as e: # connection dropped
|
||||
debug_log(f"session {self.sid} receive ended: {type(e).__name__}", "voice_worker")
|
||||
finally:
|
||||
self.closed = True
|
||||
ticker.cancel()
|
||||
writer.cancel()
|
||||
if w.scheduler:
|
||||
w.scheduler.drop_partials(lambda key: key[0] == self.sid)
|
||||
if w.tts:
|
||||
for req in list(self.tts_requests.values()):
|
||||
w.tts.cancel(req)
|
||||
debug_log(f"session {self.sid} closed", "voice_worker")
|
||||
|
||||
async def _ticker(self) -> None:
|
||||
while True:
|
||||
await asyncio.sleep(self.worker.tick_s)
|
||||
now = time.monotonic()
|
||||
for speaker, st in list(self.speakers.items()):
|
||||
self._handle_events(speaker, st, st.segmenter.tick(now))
|
||||
|
||||
# --- STT ------------------------------------------------------------------------------------
|
||||
def _on_binary(self, frame: bytes) -> None:
|
||||
if not frame or frame[0] != KIND_AUDIO:
|
||||
self.error("bad_frame", "unknown binary frame kind")
|
||||
return
|
||||
try:
|
||||
speaker, pcm = decode_audio_frame(frame)
|
||||
except ProtocolError as e:
|
||||
self.error("bad_frame", str(e))
|
||||
return
|
||||
st = self.speakers.get(speaker)
|
||||
if st is None:
|
||||
if len(self.speakers) >= MAX_SPEAKERS:
|
||||
self.error("bad_frame", f"more than {MAX_SPEAKERS} speakers")
|
||||
return
|
||||
st = _Speaker(InputConverter(self.audio["rate"], self.audio["channels"]),
|
||||
Segmenter(self.worker.vad_factory(), self.settings))
|
||||
self.speakers[speaker] = st
|
||||
samples = st.converter.push(pcm)
|
||||
self._handle_events(speaker, st, st.segmenter.push(samples, time.monotonic()))
|
||||
|
||||
def _opts(self, beam: int) -> DecodeOptions:
|
||||
return DecodeOptions(beam=beam, language=self.settings.language, hotwords=self.settings.hotwords)
|
||||
|
||||
def _handle_events(self, speaker: str, st: _Speaker, events: list) -> None:
|
||||
sched = self.worker.scheduler
|
||||
for ev in events:
|
||||
if isinstance(ev, SpeechStart):
|
||||
self.send({"type": "speech_start", "speaker": speaker, "utt": ev.utt})
|
||||
elif isinstance(ev, Partial):
|
||||
if sched and self.settings.partials:
|
||||
sched.submit_partial((self.sid, speaker), ev.audio, self._opts(1),
|
||||
self._partial_cb(speaker, st, ev.utt))
|
||||
elif isinstance(ev, Final):
|
||||
debug_log(f"utterance end speaker={speaker} utt={ev.utt} reason={ev.reason} "
|
||||
f"ms={len(ev.audio) * 1000 // SR}", "voice_worker")
|
||||
if sched is None:
|
||||
self._send_final(speaker, ev, SttResult("", 0, "stt unavailable"))
|
||||
else:
|
||||
sched.submit_final((self.sid, speaker), ev.audio, self._opts(self.settings.final_beam),
|
||||
lambda res, ev=ev: self._send_final(speaker, ev, res))
|
||||
|
||||
def _partial_cb(self, speaker: str, st: _Speaker, utt: int):
|
||||
def cb(res: SttResult) -> None:
|
||||
# Drop it when the utterance already ended or captions were turned off meanwhile.
|
||||
if st.segmenter.open_utt == utt and self.settings.partials and not res.error:
|
||||
self.send({"type": "partial", "speaker": speaker, "utt": utt, "text": res.text})
|
||||
return cb
|
||||
|
||||
def _send_final(self, speaker: str, ev: Final, res: SttResult) -> None:
|
||||
self.send({"type": "final", "speaker": speaker, "utt": ev.utt, "text": res.text,
|
||||
"end_reason": ev.reason, "audio_ms": len(ev.audio) * 1000 // SR,
|
||||
"wait_ms": int((time.monotonic() - ev.decided_at) * 1000), "stt_ms": res.stt_ms})
|
||||
|
||||
# --- text messages --------------------------------------------------------------------------
|
||||
def _on_text(self, raw: str) -> None:
|
||||
try:
|
||||
msg = json.loads(raw)
|
||||
except ValueError:
|
||||
self.error("bad_message", "not JSON")
|
||||
return
|
||||
if not isinstance(msg, dict):
|
||||
self.error("bad_message", "not an object")
|
||||
return
|
||||
kind = msg.get("type")
|
||||
handler = {"speaking_end": self._on_speaking_end, "config": self._on_config, "tts": self._on_tts,
|
||||
"tts_cancel": self._on_tts_cancel}.get(kind)
|
||||
if handler is None:
|
||||
self.error("bad_message", f"unknown type: {kind!r}")
|
||||
return
|
||||
handler(msg)
|
||||
|
||||
def _on_speaking_end(self, msg: dict) -> None:
|
||||
st = self.speakers.get(msg.get("speaker"))
|
||||
if st is not None:
|
||||
self._handle_events(msg["speaker"], st, st.segmenter.speaking_end(time.monotonic()))
|
||||
|
||||
def _on_config(self, msg: dict) -> None:
|
||||
changes = {k: v for k, v in msg.items() if k != "type"}
|
||||
try:
|
||||
self.settings.update(changes)
|
||||
except ValueError as e:
|
||||
self.error("bad_message", str(e))
|
||||
return
|
||||
if not self.settings.partials and self.worker.scheduler:
|
||||
self.worker.scheduler.drop_partials(lambda key: key[0] == self.sid)
|
||||
debug_log(f"session {self.sid} settings changed: {sorted(changes)}", "voice_worker")
|
||||
|
||||
def _on_tts(self, msg: dict) -> None:
|
||||
rid, text, more = msg.get("id"), msg.get("text", ""), msg.get("more", False)
|
||||
if not isinstance(rid, str) or not 0 < len(rid.encode()) < 256 or not isinstance(text, str) \
|
||||
or not isinstance(more, bool):
|
||||
self.error("bad_message", "tts needs a string id and text, and a boolean more")
|
||||
return
|
||||
tts = self.worker.tts
|
||||
if tts is None:
|
||||
self.error("tts_unavailable", "no TTS engine is loaded", id=rid)
|
||||
return
|
||||
req = self.tts_requests.get(rid)
|
||||
if req is None:
|
||||
req = TtsRequest(rid, self.tts_format["rate"], self._tts_chunk_cb(rid), self._tts_done,
|
||||
self._tts_error)
|
||||
self.tts_requests[rid] = req
|
||||
elif req.closed:
|
||||
self.error("bad_message", "this tts id is already closed", id=rid)
|
||||
return
|
||||
tts.feed(req, text, more)
|
||||
|
||||
def _on_tts_cancel(self, msg: dict) -> None:
|
||||
req = self.tts_requests.get(msg.get("id"))
|
||||
if req is not None and self.worker.tts:
|
||||
self.worker.tts.cancel(req)
|
||||
|
||||
def _tts_chunk_cb(self, rid: str):
|
||||
def cb(seq: int, pcm: np.ndarray) -> None:
|
||||
pcm = np.asarray(pcm, dtype="<i2")
|
||||
if self.tts_format["channels"] == 2:
|
||||
pcm = np.repeat(pcm, 2)
|
||||
self.send(encode_tts_frame(rid, seq, pcm.tobytes()))
|
||||
return cb
|
||||
|
||||
def _tts_done(self, req: TtsRequest) -> None:
|
||||
self.tts_requests.pop(req.id, None)
|
||||
self.send({"type": "tts_done", "id": req.id, "chunks": req.chunks, "cancelled": req.cancelled,
|
||||
"failed": req.failed})
|
||||
|
||||
def _tts_error(self, req: TtsRequest, message: str) -> None:
|
||||
self.error("tts_failed", message, id=req.id)
|
||||
60
src/voice_worker/settings.py
Normal file
60
src/voice_worker/settings.py
Normal file
@@ -0,0 +1,60 @@
|
||||
"""Per-connection STT settings, validated as a whole before any of them change."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass, fields
|
||||
|
||||
|
||||
@dataclass
|
||||
class SttSettings:
|
||||
partials: bool = True
|
||||
partial_interval_ms: int = 400
|
||||
end_silence_ms: int = 300
|
||||
final_beam: int = 2
|
||||
language: str | None = "ko"
|
||||
hotwords: tuple[str, ...] = ()
|
||||
|
||||
def update(self, changes: dict) -> None:
|
||||
"""Apply ``changes``; raises ValueError and changes nothing if any value is bad."""
|
||||
names = {f.name for f in fields(self)}
|
||||
staged = asdict(self)
|
||||
for key, value in changes.items():
|
||||
if key not in names:
|
||||
raise ValueError(f"unknown STT setting: {key}")
|
||||
staged[key] = _check(key, value)
|
||||
for key, value in staged.items():
|
||||
setattr(self, key, value)
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
d = asdict(self)
|
||||
d["hotwords"] = list(self.hotwords)
|
||||
return d
|
||||
|
||||
|
||||
def _int_in(key: str, value, lo: int, hi: int) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int) or not lo <= value <= hi:
|
||||
raise ValueError(f"{key} must be an integer in {lo}-{hi}")
|
||||
return value
|
||||
|
||||
|
||||
def _check(key: str, value):
|
||||
if key == "partials":
|
||||
if not isinstance(value, bool):
|
||||
raise ValueError("partials must be true or false")
|
||||
return value
|
||||
if key == "partial_interval_ms":
|
||||
return _int_in(key, value, 100, 5000)
|
||||
if key == "end_silence_ms":
|
||||
return _int_in(key, value, 100, 2000)
|
||||
if key == "final_beam":
|
||||
return _int_in(key, value, 1, 5)
|
||||
if key == "language":
|
||||
if value is not None and (not isinstance(value, str) or not 2 <= len(value) <= 8):
|
||||
raise ValueError("language must be a language code or null")
|
||||
return value
|
||||
if key == "hotwords":
|
||||
if (not isinstance(value, (list, tuple)) or len(value) > 50
|
||||
or not all(isinstance(w, str) and 0 < len(w) <= 64 for w in value)):
|
||||
raise ValueError("hotwords must be a list of up to 50 short strings")
|
||||
return tuple(value)
|
||||
raise ValueError(f"unknown STT setting: {key}") # pragma: no cover
|
||||
78
src/voice_worker/stt_whisper.py
Normal file
78
src/voice_worker/stt_whisper.py
Normal file
@@ -0,0 +1,78 @@
|
||||
"""faster-whisper batch transcription: one encoder and one decoder pass for every clip in a batch."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
from .audio import peak_normalise
|
||||
from .scheduler import DecodeOptions
|
||||
|
||||
SR = 16000
|
||||
NO_SPEECH_THRESHOLD = 0.5 # same gate as v1 (whisper_no_speech_threshold)
|
||||
MIN_CONFIDENCE = 0.3 # avg_logprob + 1, same as v1 (whisper_min_confidence)
|
||||
|
||||
|
||||
def add_cuda_dll_dirs() -> None:
|
||||
"""On Windows ctranslate2 loads cuBLAS/cuDNN from the PATH; the pip wheels put them in nvidia/*/bin."""
|
||||
if not hasattr(os, "add_dll_directory"):
|
||||
return
|
||||
for sp in sys.path:
|
||||
for d in glob.glob(os.path.join(sp, "nvidia", "*", "bin")):
|
||||
if os.path.isdir(d):
|
||||
os.environ["PATH"] = d + os.pathsep + os.environ.get("PATH", "")
|
||||
os.add_dll_directory(d)
|
||||
|
||||
|
||||
class WhisperTranscriber:
|
||||
def __init__(self, model: str = "large-v3-turbo", device: str = "cuda", compute_type: str = "int8_float16"):
|
||||
add_cuda_dll_dirs()
|
||||
from faster_whisper import WhisperModel
|
||||
self.model = WhisperModel(model, device=device, compute_type=compute_type)
|
||||
self._tokenizers: dict = {}
|
||||
debug_log(f"whisper loaded model={model} device={device} compute={compute_type}", "voice_worker")
|
||||
|
||||
def _tokenizer(self, language: str):
|
||||
tok = self._tokenizers.get(language)
|
||||
if tok is None:
|
||||
from faster_whisper.tokenizer import Tokenizer
|
||||
tok = Tokenizer(self.model.hf_tokenizer, self.model.model.is_multilingual, task="transcribe",
|
||||
language=language)
|
||||
self._tokenizers[language] = tok
|
||||
return tok
|
||||
|
||||
def warm_up(self) -> None:
|
||||
self.transcribe([np.zeros(SR, dtype=np.float32)], DecodeOptions(beam=1, language="en", hotwords=()))
|
||||
|
||||
def transcribe(self, clips: list[np.ndarray], opts: DecodeOptions) -> list[str]:
|
||||
m = self.model
|
||||
n_frames = m.feature_extractor.nb_max_frames
|
||||
feats = []
|
||||
for c in clips:
|
||||
a = peak_normalise(c[: SR * 30])
|
||||
a = np.pad(a, (0, SR * 30 - len(a)))
|
||||
feats.append(m.feature_extractor(a, padding=0)[:, :n_frames])
|
||||
enc = m.encode(np.stack(feats).astype(np.float32))
|
||||
if opts.language:
|
||||
langs = [opts.language] * len(clips)
|
||||
else:
|
||||
langs = [r[0][0][2:-2] for r in m.model.detect_language(enc)]
|
||||
hot = " ".join(opts.hotwords) or None
|
||||
prompts = [m.get_prompt(self._tokenizer(lang), [], without_timestamps=True, hotwords=hot) for lang in langs]
|
||||
results = m.model.generate(enc, prompts, beam_size=opts.beam, max_length=224,
|
||||
suppress_blank=True, suppress_tokens=[-1], return_scores=True,
|
||||
return_no_speech_prob=True)
|
||||
texts = []
|
||||
for lang, r in zip(langs, results):
|
||||
tokens = r.sequences_ids[0]
|
||||
avg_logprob = r.scores[0] * len(tokens) / (len(tokens) + 1)
|
||||
if r.no_speech_prob >= NO_SPEECH_THRESHOLD or avg_logprob + 1 < MIN_CONFIDENCE:
|
||||
texts.append("")
|
||||
else:
|
||||
texts.append(self._tokenizer(lang).decode(tokens).strip())
|
||||
return texts
|
||||
105
src/voice_worker/tts.py
Normal file
105
src/voice_worker/tts.py
Normal file
@@ -0,0 +1,105 @@
|
||||
"""Sentence-by-sentence TTS shared by every connection, first come first served."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Protocol
|
||||
|
||||
import numpy as np
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
from .sentences import StreamingSplitter
|
||||
|
||||
|
||||
class TtsEngine(Protocol):
|
||||
def synthesize(self, text: str, rate: int) -> np.ndarray:
|
||||
"""Return int16 mono samples at ``rate``."""
|
||||
...
|
||||
|
||||
|
||||
@dataclass
|
||||
class TtsRequest:
|
||||
id: str
|
||||
rate: int
|
||||
on_chunk: Callable[[int, np.ndarray], None]
|
||||
on_done: Callable[["TtsRequest"], None]
|
||||
on_error: Callable[["TtsRequest", str], None]
|
||||
splitter: StreamingSplitter = field(default_factory=StreamingSplitter)
|
||||
next_seq: int = 0
|
||||
outstanding: int = 0
|
||||
chunks: int = 0
|
||||
closed: bool = False
|
||||
cancelled: bool = False
|
||||
failed: bool = False
|
||||
finished: bool = False
|
||||
|
||||
|
||||
class TtsService:
|
||||
def __init__(self, engine: TtsEngine):
|
||||
self.engine = engine
|
||||
self._queue: asyncio.Queue = asyncio.Queue()
|
||||
self._executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="tts")
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
self._task = asyncio.get_running_loop().create_task(self._run())
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._executor.shutdown(wait=False, cancel_futures=True)
|
||||
|
||||
def feed(self, req: TtsRequest, text: str, more: bool) -> None:
|
||||
sentences = req.splitter.feed(text)
|
||||
if not more:
|
||||
sentences += req.splitter.finish()
|
||||
req.closed = True
|
||||
for s in sentences:
|
||||
req.outstanding += 1
|
||||
self._queue.put_nowait((req, req.next_seq, s))
|
||||
req.next_seq += 1
|
||||
self._maybe_done(req)
|
||||
|
||||
def cancel(self, req: TtsRequest) -> None:
|
||||
req.cancelled = True
|
||||
req.closed = True
|
||||
debug_log(f"tts cancel id={req.id} sent={req.chunks}", "voice_worker")
|
||||
self._maybe_done(req)
|
||||
|
||||
def _maybe_done(self, req: TtsRequest) -> None:
|
||||
if req.closed and req.outstanding == 0 and not req.finished:
|
||||
req.finished = True
|
||||
req.on_done(req)
|
||||
|
||||
async def _run(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
while True:
|
||||
req, seq, text = await self._queue.get()
|
||||
try:
|
||||
if req.cancelled or req.failed:
|
||||
continue
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
pcm = await loop.run_in_executor(self._executor, self.engine.synthesize, text, req.rate)
|
||||
except Exception as e:
|
||||
req.failed = True
|
||||
req.closed = True
|
||||
debug_log(f"tts failed id={req.id} seq={seq}: {type(e).__name__}: {e}", "voice_worker")
|
||||
req.on_error(req, f"{type(e).__name__}: {e}")
|
||||
continue
|
||||
debug_log(f"tts sentence id={req.id} seq={seq} chars={len(text)} "
|
||||
f"ms={int((time.perf_counter() - t0) * 1000)}", "voice_worker")
|
||||
if not req.cancelled:
|
||||
req.chunks += 1
|
||||
req.on_chunk(seq, pcm)
|
||||
finally:
|
||||
req.outstanding -= 1
|
||||
self._maybe_done(req)
|
||||
65
src/voice_worker/tts_melo.py
Normal file
65
src/voice_worker/tts_melo.py
Normal file
@@ -0,0 +1,65 @@
|
||||
"""TTS engine that drives ``melo_child.py`` in the MeloTTS venv over stdin/stdout."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
|
||||
import numpy as np
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
CHILD = os.path.join(os.path.dirname(os.path.abspath(__file__)), "melo_child.py")
|
||||
|
||||
|
||||
class MeloEngine:
|
||||
def __init__(self, python: str, device: str = "cuda", speed: float = 1.5, language: str = "KR"):
|
||||
self.cmd = [python, CHILD, "--device", device, "--speed", str(speed), "--language", language]
|
||||
self._lock = threading.Lock()
|
||||
self._proc: subprocess.Popen | None = None
|
||||
self.native_rate: int | None = None
|
||||
self._start()
|
||||
|
||||
def _start(self) -> None:
|
||||
self._proc = subprocess.Popen(self.cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
|
||||
stderr=None, creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
|
||||
header, _ = self._read()
|
||||
if not header.get("ready"):
|
||||
self.close()
|
||||
raise RuntimeError(f"MeloTTS failed to start: {header.get('error')}")
|
||||
self.native_rate = header["rate"]
|
||||
debug_log(f"melo ready native_rate={self.native_rate}", "voice_worker")
|
||||
|
||||
def _read(self) -> tuple[dict, bytes]:
|
||||
out = self._proc.stdout
|
||||
raw = out.read(4)
|
||||
if len(raw) < 4:
|
||||
raise RuntimeError("MeloTTS child exited")
|
||||
(n,) = struct.unpack("<I", raw)
|
||||
header = json.loads(out.read(n))
|
||||
payload = out.read(header.get("samples", 0) * 2) if header.get("ok") else b""
|
||||
return header, payload
|
||||
|
||||
def synthesize(self, text: str, rate: int) -> np.ndarray:
|
||||
with self._lock:
|
||||
if self._proc is None or self._proc.poll() is not None:
|
||||
debug_log("melo child not running, restarting", "voice_worker")
|
||||
self._start()
|
||||
self._proc.stdin.write((json.dumps({"text": text, "rate": rate}) + "\n").encode())
|
||||
self._proc.stdin.flush()
|
||||
header, payload = self._read()
|
||||
if not header.get("ok"):
|
||||
raise RuntimeError(header.get("error", "MeloTTS failed"))
|
||||
return np.frombuffer(payload, dtype="<i2")
|
||||
|
||||
def close(self) -> None:
|
||||
if self._proc and self._proc.poll() is None:
|
||||
self._proc.stdin.close()
|
||||
try:
|
||||
self._proc.wait(5)
|
||||
except subprocess.TimeoutExpired:
|
||||
self._proc.kill()
|
||||
45
src/voice_worker/vad.py
Normal file
45
src/voice_worker/vad.py
Normal file
@@ -0,0 +1,45 @@
|
||||
"""Streaming Silero VAD (the v6 model bundled with faster-whisper), one state per speaker."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .segmenter import WINDOW
|
||||
|
||||
_CONTEXT = 64
|
||||
|
||||
|
||||
class SileroVad:
|
||||
"""Owns the ONNX session; ``stream()`` gives each speaker its own recurrent state."""
|
||||
|
||||
def __init__(self, path: str | None = None):
|
||||
import onnxruntime
|
||||
if path is None:
|
||||
from faster_whisper.utils import get_assets_path
|
||||
path = os.path.join(get_assets_path(), "silero_vad_v6.onnx")
|
||||
opts = onnxruntime.SessionOptions()
|
||||
opts.inter_op_num_threads = 1
|
||||
opts.intra_op_num_threads = 1
|
||||
opts.enable_cpu_mem_arena = False
|
||||
opts.log_severity_level = 4
|
||||
self.session = onnxruntime.InferenceSession(path, providers=["CPUExecutionProvider"], sess_options=opts)
|
||||
|
||||
def stream(self) -> "SileroStream":
|
||||
return SileroStream(self.session)
|
||||
|
||||
|
||||
class SileroStream:
|
||||
def __init__(self, session):
|
||||
self.session = session
|
||||
self.h = np.zeros((1, 1, 128), dtype=np.float32)
|
||||
self.c = np.zeros((1, 1, 128), dtype=np.float32)
|
||||
self.context = np.zeros(_CONTEXT, dtype=np.float32)
|
||||
|
||||
def prob(self, window: np.ndarray) -> float:
|
||||
assert len(window) == WINDOW
|
||||
x = np.concatenate([self.context, window.astype(np.float32, copy=False)])[None, :]
|
||||
out, self.h, self.c = self.session.run(None, {"input": x, "h": self.h, "c": self.c})
|
||||
self.context = x[0, -_CONTEXT:]
|
||||
return float(np.asarray(out).reshape(-1)[0])
|
||||
148
src/voice_worker/voice_worker.spec.md
Normal file
148
src/voice_worker/voice_worker.spec.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Voice worker spec (v2 M1)
|
||||
|
||||
The voice worker is the Python process that turns many Discord speakers' audio into text and the
|
||||
assistant's reply text into speech. The Electron app's Discord voice core talks to it over one local
|
||||
WebSocket. Design background lives in `docs/v2/voice-and-llm.md` and `docs/v2/architecture.md`.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Local and private.** The server binds to a loopback address only and refuses any other host. Every
|
||||
connection must present the token the app generated; the token comes from the environment
|
||||
(`JARVIS_VOICE_TOKEN`), never from the command line, so it does not show in process listings. Logs
|
||||
carry timings, sizes and counts, never transcript or reply text.
|
||||
- **One shared model.** A single faster-whisper model serves every speaker. Speakers that need
|
||||
transcription at the same moment go through the model in one batch.
|
||||
- **Finals before partials.** A finished utterance is always transcribed before any live caption.
|
||||
- **Language-agnostic.** Nothing in the worker matches words. The STT language is a setting (`null`
|
||||
means detect per utterance); sentence splitting uses punctuation only.
|
||||
- **Engines are pluggable.** STT, VAD and TTS sit behind small interfaces so the logic is tested with
|
||||
fakes and the real models are tested separately.
|
||||
|
||||
## Running
|
||||
|
||||
```
|
||||
JARVIS_VOICE_TOKEN=<secret> python -m voice_worker --host 127.0.0.1 --port 8771 \
|
||||
[--stt-model large-v3-turbo] [--stt-device cuda] [--stt-compute int8_float16] \
|
||||
[--melo-python <path to the MeloTTS venv python>] [--melo-device cuda] [--melo-speed 1.5]
|
||||
```
|
||||
|
||||
Run from `src/` (or with `src/` on `PYTHONPATH`) in a venv built from `voice_worker/requirements.txt`. MeloTTS needs its own Python 3.11 venv
|
||||
(transformers 4.27), so it runs as a child process (`voice_worker/melo_child.py`) under
|
||||
`--melo-python`. Without `--melo-python` the worker has no TTS and answers `tts` requests with an
|
||||
error. Both models load and warm up once at start; `hello_ok` reports whether each is ready.
|
||||
|
||||
## Protocol
|
||||
|
||||
Text frames are JSON objects with a `type`. Binary frames start with a one-byte kind.
|
||||
|
||||
### Handshake
|
||||
|
||||
The first frame must be `hello` within 5 seconds, otherwise the worker closes with code 4400. A wrong
|
||||
token closes with 4401.
|
||||
|
||||
```
|
||||
-> {"type": "hello", "token": "...",
|
||||
"audio": {"rate": 48000, "channels": 2}, // incoming PCM format: rate 16000|48000, channels 1|2
|
||||
"tts": {"rate": 48000, "channels": 2}, // outgoing PCM format (any rate, channels 1|2)
|
||||
"stt": { ...settings, see config... }} // optional
|
||||
<- {"type": "hello_ok", "stt_ready": true, "tts_ready": true,
|
||||
"audio": {...}, "tts": {...}, "stt": { ...effective settings... }}
|
||||
```
|
||||
|
||||
### STT
|
||||
|
||||
```
|
||||
-> binary 0x01 | u8 n | speaker id (n bytes UTF-8) | PCM16LE samples in the hello "audio" format
|
||||
-> {"type": "speaking_end", "speaker": "..."} // Discord speaking stopped for this speaker
|
||||
-> {"type": "config", ...any STT settings...} // takes effect immediately
|
||||
<- {"type": "speech_start", "speaker": "...", "utt": 3}
|
||||
<- {"type": "partial", "speaker": "...", "utt": 3, "text": "..."}
|
||||
<- {"type": "final", "speaker": "...", "utt": 3, "text": "...", "end_reason": "silence",
|
||||
"audio_ms": 2310, "wait_ms": 160, "stt_ms": 120}
|
||||
```
|
||||
|
||||
`utt` counts utterances per speaker. `final` always closes an utterance that sent `speech_start`, even
|
||||
when the text is empty (noise that Whisper rejected). `wait_ms` is from the moment the end was decided
|
||||
to the moment the final was sent; `stt_ms` is the model time of the batch it went through.
|
||||
|
||||
STT settings (all optional in `hello` and `config`):
|
||||
|
||||
| Setting | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `partials` | `true` | Send live captions. `false` drops queued partials and stops new ones; finals continue. The app turns it off while the LLM answers during a game. |
|
||||
| `partial_interval_ms` | 400 | At most one partial per speaker per interval, only when audio grew. |
|
||||
| `end_silence_ms` | 300 | Trailing VAD silence that ends an utterance. Range 100-2000. |
|
||||
| `final_beam` | 2 | Beam size for finals (partials always use 1). Range 1-5. |
|
||||
| `language` | `"ko"` | Whisper language code, or `null` to detect per utterance. |
|
||||
| `hotwords` | `[]` | Words (wake words, names) passed to Whisper as a hint. |
|
||||
|
||||
### Utterance rules
|
||||
|
||||
Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filter, and run through
|
||||
its own streaming Silero VAD (512-sample windows, state kept between frames).
|
||||
|
||||
- **Start**: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
|
||||
- **speech_start** is sent once the utterance holds 250 ms of speech windows. An utterance that ends
|
||||
before that is dropped without any message (a click or a cough).
|
||||
- **Speech continues** while windows stay at or above 0.35 (hysteresis); a quieter window counts as
|
||||
silence.
|
||||
- **End**, with `end_reason`:
|
||||
- `silence`: trailing silence reached `end_silence_ms`.
|
||||
- `speaking_end`: the app sent `speaking_end`. Discord stops sending packets when a user stops
|
||||
talking, so the worker cannot wait for silence that never arrives.
|
||||
- `idle`: no audio arrived for `end_silence_ms` (a missed `speaking_end`).
|
||||
- `max_length`: the utterance reached 28 seconds (Whisper's window). The next audio starts a new
|
||||
utterance.
|
||||
- The final clip keeps at most 200 ms of trailing silence.
|
||||
- Before transcription the clip is peak-normalised to 0.9 with the gain capped at 10x.
|
||||
|
||||
### Scheduling
|
||||
|
||||
One thread owns the STT model and loops:
|
||||
|
||||
1. If any finals are waiting, take all of them (up to 16) and transcribe in one batch with
|
||||
`final_beam`.
|
||||
2. Otherwise take up to 2 waiting partials and transcribe them in one batch with beam 1. A speaker
|
||||
has at most one waiting partial; a newer request replaces the older one and goes to the back of the
|
||||
line, so speakers take turns.
|
||||
|
||||
Whisper's encoder always runs on a full 30-second window, so its cost grows with every clip in a batch
|
||||
(measured on an RTX 5050: 0.2 s for one clip, 2.1 s for ten). A large partial batch would make a final
|
||||
wait seconds behind it; two clips keep that wait to about one clip's time. With many people talking,
|
||||
captions fall behind and only the newest audio per speaker is transcribed.
|
||||
|
||||
Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has
|
||||
already ended, or that is older than the last partial sent, is dropped. Whisper output is rejected
|
||||
(empty text) when `no_speech_prob >= 0.5` or the confidence (`avg_logprob + 1`) is below 0.3, the
|
||||
same gate as v1.
|
||||
|
||||
### TTS
|
||||
|
||||
```
|
||||
-> {"type": "tts", "id": "r1", "text": "...", "more": false} // more=true: more text follows for r1
|
||||
-> {"type": "tts_cancel", "id": "r1"}
|
||||
<- binary 0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
|
||||
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}
|
||||
```
|
||||
|
||||
Text is split into sentences on punctuation only: a run of newlines, CJK terminators (`。!?…`), or
|
||||
ASCII terminators followed by whitespace, a closing quote or bracket, or the end of the text, so
|
||||
decimals, versions and URLs stay whole. Fragments under 5 characters are merged into a neighbour.
|
||||
With `more=true` the worker synthesises every complete sentence at once and holds the unfinished tail
|
||||
until more text or `more=false` arrives (a short last fragment is then spoken on its own, because the
|
||||
sentence before it has already gone out). One audio chunk is sent per sentence, in order, as soon as it
|
||||
is ready. Requests are served first come, first served by one TTS engine. `tts_cancel` drops the
|
||||
request's unsent sentences, and `tts_done` follows with `cancelled: true`. A sentence that fails to
|
||||
synthesise is reported with `error` (`code: "tts_failed"`) and the request ends with `failed: true`.
|
||||
|
||||
### Errors
|
||||
|
||||
`{"type": "error", "code": "...", "message": "...", "id": "..."}`. Codes: `bad_message` (unknown type,
|
||||
bad JSON, bad field), `bad_frame` (malformed binary frame), `tts_unavailable`, `tts_failed`. Errors
|
||||
never close the connection after the handshake.
|
||||
|
||||
## Debug logging
|
||||
|
||||
`debug_log(..., "voice_worker")` at: connection accepted or refused, settings changed, utterance end
|
||||
(speaker, reason, length), each STT batch (kind, size, model time), each TTS sentence (length, time),
|
||||
cancel, and engine failures.
|
||||
122
tests/test_voice_worker_audio.py
Normal file
122
tests/test_voice_worker_audio.py
Normal file
@@ -0,0 +1,122 @@
|
||||
"""Voice worker: binary frames, input conversion to 16 kHz mono, clip normalisation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from voice_worker.audio import InputConverter, peak_normalise
|
||||
from voice_worker.protocol import (KIND_AUDIO, KIND_TTS, ProtocolError, decode_audio_frame, encode_audio_frame,
|
||||
encode_tts_frame, decode_tts_frame)
|
||||
|
||||
|
||||
def _pcm16(x: np.ndarray) -> bytes:
|
||||
return (np.clip(x, -1, 1) * 32767).astype("<i2").tobytes()
|
||||
|
||||
|
||||
def _dominant_hz(x: np.ndarray, rate: int) -> float:
|
||||
spec = np.abs(np.fft.rfft(x * np.hanning(len(x))))
|
||||
return float(np.fft.rfftfreq(len(x), 1 / rate)[int(np.argmax(spec))])
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_audio_frame_round_trip_keeps_speaker_and_pcm():
|
||||
pcm = _pcm16(np.linspace(-0.5, 0.5, 960))
|
||||
frame = encode_audio_frame("123456789012345678", pcm)
|
||||
assert frame[0] == KIND_AUDIO
|
||||
speaker, payload = decode_audio_frame(frame)
|
||||
assert speaker == "123456789012345678"
|
||||
assert payload == pcm
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_tts_frame_round_trip_keeps_id_and_sequence():
|
||||
pcm = b"\x01\x00\x02\x00"
|
||||
frame = encode_tts_frame("reply-7", 3, pcm)
|
||||
assert frame[0] == KIND_TTS
|
||||
assert decode_tts_frame(frame) == ("reply-7", 3, pcm)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("frame", [b"", b"\x01", b"\x09\x01a\x00\x00", b"\x01\x05ab", b"\x01\x01a\x00",
|
||||
b"\x01\x02\xff\xfe\x00\x00"])
|
||||
def test_malformed_audio_frames_are_rejected(frame):
|
||||
with pytest.raises(ProtocolError):
|
||||
decode_audio_frame(frame)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_48k_stereo_becomes_16k_mono_and_keeps_the_tone():
|
||||
rate = 48000
|
||||
t = np.arange(rate) / rate
|
||||
tone = 0.5 * np.sin(2 * np.pi * 1000 * t)
|
||||
stereo = np.stack([tone, tone], axis=1).reshape(-1)
|
||||
out = InputConverter(rate, 2).push(_pcm16(stereo))
|
||||
assert out.dtype == np.float32
|
||||
assert abs(len(out) - rate // 3) <= 1
|
||||
assert abs(_dominant_hz(out, 16000) - 1000) < 5
|
||||
assert 0.4 < np.max(np.abs(out[500:])) < 0.6
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_tones_above_the_new_nyquist_are_filtered_out():
|
||||
rate = 48000
|
||||
t = np.arange(rate) / rate
|
||||
high = 0.5 * np.sin(2 * np.pi * 11000 * t)
|
||||
out = InputConverter(rate, 1).push(_pcm16(high))
|
||||
# At least 30 dB down instead of folding back to 5 kHz.
|
||||
assert np.sqrt(np.mean(out[500:] ** 2)) < 0.5 / np.sqrt(2) * 10 ** (-30 / 20)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_streaming_in_20ms_frames_matches_one_shot_conversion():
|
||||
rng = np.random.default_rng(1)
|
||||
sig = rng.uniform(-0.5, 0.5, 48000 * 2 // 2 * 2)
|
||||
one = InputConverter(48000, 2).push(_pcm16(sig))
|
||||
conv = InputConverter(48000, 2)
|
||||
raw = _pcm16(sig)
|
||||
step = 960 * 2 * 2 # 20 ms of 48 kHz stereo PCM16
|
||||
parts = [conv.push(raw[i:i + step]) for i in range(0, len(raw), step)]
|
||||
assert np.allclose(np.concatenate(parts), one, atol=1e-6)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_16k_mono_passes_through():
|
||||
sig = np.linspace(-0.25, 0.25, 1600)
|
||||
out = InputConverter(16000, 1).push(_pcm16(sig))
|
||||
assert np.allclose(out, sig, atol=1e-4)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_unsupported_input_formats_are_refused():
|
||||
with pytest.raises(ValueError):
|
||||
InputConverter(44100, 2)
|
||||
with pytest.raises(ValueError):
|
||||
InputConverter(48000, 6)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_odd_byte_counts_are_carried_to_the_next_frame():
|
||||
sig = _pcm16(np.full(10, 0.5))
|
||||
conv = InputConverter(16000, 1)
|
||||
a = conv.push(sig[:5])
|
||||
b = conv.push(sig[5:])
|
||||
assert len(a) + len(b) == 10
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_quiet_clip_is_raised_to_the_target_peak():
|
||||
x = np.full(100, 0.2, dtype=np.float32)
|
||||
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_near_silence_is_not_boosted_past_the_gain_cap():
|
||||
x = np.full(100, 0.001, dtype=np.float32)
|
||||
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.01)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_loud_clip_is_brought_down_to_the_target_peak():
|
||||
x = np.array([0.0, 1.0, -0.5], dtype=np.float32)
|
||||
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)
|
||||
209
tests/test_voice_worker_real.py
Normal file
209
tests/test_voice_worker_real.py
Normal file
@@ -0,0 +1,209 @@
|
||||
"""Voice worker with the real models: ten recorded clips streamed at once, and a TTS round trip.
|
||||
|
||||
Opt-in because it needs a CUDA GPU and the downloaded models:
|
||||
|
||||
JARVIS_VOICE_REAL_TESTS=1 [JARVIS_MELO_PYTHON=<MeloTTS venv python>] pytest tests/test_voice_worker_real.py -s
|
||||
|
||||
On Linux, put the pip CUDA libraries on LD_LIBRARY_PATH (nvidia/cublas/lib, nvidia/cudnn/lib).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
import wave
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
pytestmark = [pytest.mark.integration,
|
||||
pytest.mark.skipif(os.environ.get("JARVIS_VOICE_REAL_TESTS") != "1",
|
||||
reason="set JARVIS_VOICE_REAL_TESTS=1 to run on the GPU")]
|
||||
|
||||
SAMPLES = Path(__file__).resolve().parent.parent / "scripts" / "m0" / "assets" / "ko_samples"
|
||||
TOKEN = "real-test-token"
|
||||
FRAME_MS = 20
|
||||
|
||||
|
||||
def cer(ref: str, hyp: str) -> float:
|
||||
a, b = (re.sub(r"[\W_]+", "", s) for s in (ref, hyp))
|
||||
prev = list(range(len(b) + 1))
|
||||
for i, ca in enumerate(a, 1):
|
||||
cur = [i]
|
||||
for j, cb in enumerate(b, 1):
|
||||
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
|
||||
prev = cur
|
||||
return prev[-1] / max(1, len(a))
|
||||
|
||||
|
||||
def load(name: str) -> np.ndarray:
|
||||
with wave.open(str(SAMPLES / name), "rb") as w:
|
||||
assert w.getframerate() == 16000 and w.getnchannels() == 1
|
||||
return np.frombuffer(w.readframes(w.getnframes()), dtype="<i2").astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
def to_48k_stereo(x16: np.ndarray) -> np.ndarray:
|
||||
t48 = np.arange(len(x16) * 3) / 3
|
||||
x48 = np.interp(t48, np.arange(len(x16)), x16)
|
||||
return np.repeat(x48, 2)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def engines():
|
||||
from voice_worker.stt_whisper import WhisperTranscriber
|
||||
from voice_worker.vad import SileroVad
|
||||
stt = WhisperTranscriber()
|
||||
stt.warm_up()
|
||||
tts = None
|
||||
if os.environ.get("JARVIS_MELO_PYTHON"):
|
||||
from voice_worker.tts_melo import MeloEngine
|
||||
tts = MeloEngine(os.environ["JARVIS_MELO_PYTHON"])
|
||||
yield stt, SileroVad(), tts
|
||||
if tts:
|
||||
tts.close()
|
||||
|
||||
|
||||
async def with_worker(engines, body):
|
||||
from websockets.asyncio.client import connect
|
||||
from voice_worker.server import VoiceWorker
|
||||
stt, vad, tts = engines
|
||||
worker = VoiceWorker(TOKEN, stt, vad.stream, tts)
|
||||
server = await worker.serve("127.0.0.1", 0)
|
||||
url = f"ws://127.0.0.1:{server.sockets[0].getsockname()[1]}"
|
||||
try:
|
||||
async with connect(url, max_size=2 ** 24) as ws:
|
||||
await ws.send(json.dumps({"type": "hello", "token": TOKEN, "stt": {"hotwords": ["자비스"]}}))
|
||||
assert json.loads(await ws.recv())["type"] == "hello_ok"
|
||||
return await asyncio.wait_for(body(ws), 120)
|
||||
finally:
|
||||
server.close()
|
||||
await server.wait_closed()
|
||||
await worker.stop()
|
||||
|
||||
|
||||
def test_ten_speakers_at_once(engines):
|
||||
"""Ten speakers talk over each other in real time (48 kHz stereo, 20 ms frames, staggered starts).
|
||||
Half end with Discord's speaking_end, half keep sending silence so only the VAD can end them."""
|
||||
from voice_worker.protocol import encode_audio_frame
|
||||
|
||||
refs = json.loads((SAMPLES / "refs.json").read_text(encoding="utf-8"))["samples"]
|
||||
files = [refs[i % len(refs)] for i in range(10)]
|
||||
speakers = [f"spk{i}" for i in range(10)]
|
||||
frame = 48000 * FRAME_MS // 1000 * 2 # stereo samples per frame
|
||||
|
||||
def track(i: int) -> list[bytes]:
|
||||
lead = np.zeros(int(0.15 * i * 16000), dtype=np.float32)
|
||||
tail = np.zeros(int(0.6 * 16000), dtype=np.float32) if i % 2 else np.zeros(0, dtype=np.float32)
|
||||
clip = load(files[i]["file"])
|
||||
if i % 2 == 0: # these end with speaking_end: Discord stops sending right after the last word
|
||||
loud = np.nonzero(np.abs(clip) > 0.02)[0]
|
||||
clip = clip[: loud[-1] + 800]
|
||||
x = to_48k_stereo(np.concatenate([lead, clip, tail]))
|
||||
pcm = (np.clip(x, -1, 1) * 32767).astype("<i2")
|
||||
return [pcm[k:k + frame].tobytes() for k in range(0, len(pcm), frame)]
|
||||
|
||||
tracks = [track(i) for i in range(10)]
|
||||
|
||||
async def body(ws):
|
||||
finals: dict[str, list[dict]] = {s: [] for s in speakers}
|
||||
partials = 0
|
||||
finished = asyncio.Event()
|
||||
|
||||
async def reader():
|
||||
nonlocal partials
|
||||
async for raw in ws:
|
||||
msg = json.loads(raw)
|
||||
if msg["type"] == "speech_start":
|
||||
open_utts[msg["speaker"]] = msg["utt"]
|
||||
elif msg["type"] == "partial":
|
||||
partials += 1
|
||||
elif msg["type"] == "final":
|
||||
finals[msg["speaker"]].append(msg)
|
||||
if sent_all.is_set() and all(finals.values()) and closed_all():
|
||||
finished.set()
|
||||
|
||||
open_utts: dict[str, int] = {}
|
||||
|
||||
def closed_all() -> bool:
|
||||
return all(len(finals[s]) >= open_utts.get(s, 0) for s in speakers)
|
||||
|
||||
sent_all = asyncio.Event()
|
||||
rd = asyncio.create_task(reader())
|
||||
t0 = time.monotonic()
|
||||
for k in range(max(len(t) for t in tracks)):
|
||||
for i, s in enumerate(speakers):
|
||||
if k < len(tracks[i]):
|
||||
await ws.send(encode_audio_frame(s, tracks[i][k]))
|
||||
elif k == len(tracks[i]) and i % 2 == 0:
|
||||
await ws.send(json.dumps({"type": "speaking_end", "speaker": s}))
|
||||
await asyncio.sleep(max(0.0, t0 + (k + 1) * FRAME_MS / 1000 - time.monotonic()))
|
||||
sent_all.set()
|
||||
try:
|
||||
await asyncio.wait_for(finished.wait(), 20)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
rd.cancel()
|
||||
return finals, partials
|
||||
|
||||
finals, partials = asyncio.run(with_worker(engines, body))
|
||||
rows = []
|
||||
for i, s in enumerate(speakers):
|
||||
text = " ".join(m["text"] for m in finals[s]).strip()
|
||||
rows.append((s, files[i]["file"], len(finals[s]), cer(files[i]["text"], text),
|
||||
[m["end_reason"] for m in finals[s]], [m["wait_ms"] for m in finals[s]], text))
|
||||
for r in rows:
|
||||
print(*r)
|
||||
waits = [w for r in rows for w in r[5]]
|
||||
print(f"partials={partials} wait_ms median={np.median(waits):.0f} max={max(waits)}")
|
||||
assert all(r[2] >= 1 for r in rows), "every speaker gets a final"
|
||||
assert np.mean([r[3] for r in rows]) <= 0.15
|
||||
assert max(r[3] for r in rows) <= 0.35
|
||||
assert {"speaking_end", "silence"} <= {reason for r in rows for reason in r[4]}
|
||||
assert partials > 0
|
||||
|
||||
|
||||
def test_tts_reply_comes_back_as_the_same_words(engines):
|
||||
"""Speech the worker synthesises, fed back in as a speaker, is transcribed to the same text."""
|
||||
if engines[2] is None:
|
||||
pytest.skip("set JARVIS_MELO_PYTHON to the MeloTTS venv python")
|
||||
from voice_worker.protocol import decode_tts_frame, encode_audio_frame
|
||||
text = "오늘 서울은 맑고 기온은 이십삼 도예요. 우산은 필요 없어요."
|
||||
|
||||
async def body(ws):
|
||||
await ws.send(json.dumps({"type": "config", "partials": False}))
|
||||
await ws.send(json.dumps({"type": "tts", "id": "t1", "text": text}))
|
||||
chunks = []
|
||||
t0 = time.monotonic()
|
||||
first = None
|
||||
while True:
|
||||
raw = await ws.recv()
|
||||
if isinstance(raw, bytes):
|
||||
chunks.append(decode_tts_frame(raw))
|
||||
first = first or time.monotonic() - t0
|
||||
elif json.loads(raw)["type"] == "tts_done":
|
||||
break
|
||||
pcm = b"".join(c[2] for c in chunks)
|
||||
step = 48000 * FRAME_MS // 1000 * 2 * 2
|
||||
for k in range(0, len(pcm), step):
|
||||
await ws.send(encode_audio_frame("echo", pcm[k:k + step]))
|
||||
await ws.send(json.dumps({"type": "speaking_end", "speaker": "echo"}))
|
||||
got = []
|
||||
# The pause between the sentences may split them, and the clip's own trailing silence may end
|
||||
# the last one before speaking_end does, so collect finals until the worker goes quiet.
|
||||
while True:
|
||||
try:
|
||||
msg = json.loads(await asyncio.wait_for(ws.recv(), 3))
|
||||
except asyncio.TimeoutError:
|
||||
break
|
||||
if msg["type"] == "final":
|
||||
got.append(msg["text"])
|
||||
return chunks, first, " ".join(got)
|
||||
|
||||
chunks, first, heard = asyncio.run(with_worker(engines, body))
|
||||
print(f"tts chunks={len(chunks)} first_chunk_s={first:.2f} heard={heard}")
|
||||
assert [c[1] for c in chunks] == [0, 1]
|
||||
assert cer(text, heard) <= 0.2
|
||||
253
tests/test_voice_worker_scheduler.py
Normal file
253
tests/test_voice_worker_scheduler.py
Normal file
@@ -0,0 +1,253 @@
|
||||
"""Voice worker: the shared-model STT scheduler (batching, finals first, partial coalescing)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from voice_worker.scheduler import DecodeOptions, SttScheduler
|
||||
|
||||
FINAL = DecodeOptions(beam=2, language="ko", hotwords=())
|
||||
PARTIAL = DecodeOptions(beam=1, language="ko", hotwords=())
|
||||
|
||||
|
||||
class FakeTranscriber:
|
||||
"""Names each clip by its length; the first call can be held to build up a queue."""
|
||||
|
||||
def __init__(self, hold_first: bool = False, fail_on_beam: int | None = None):
|
||||
self.calls: list[tuple[DecodeOptions, list[int]]] = []
|
||||
self.gate = threading.Event()
|
||||
self.entered = threading.Event()
|
||||
if not hold_first:
|
||||
self.gate.set()
|
||||
self.fail_on_beam = fail_on_beam
|
||||
|
||||
def transcribe(self, clips, opts):
|
||||
self.entered.set()
|
||||
self.gate.wait(5)
|
||||
self.calls.append((opts, [len(c) for c in clips]))
|
||||
if opts.beam == self.fail_on_beam:
|
||||
raise RuntimeError("boom")
|
||||
return [f"n{len(c)}" for c in clips]
|
||||
|
||||
|
||||
def clip(n: int) -> np.ndarray:
|
||||
return np.zeros(n, dtype=np.float32)
|
||||
|
||||
|
||||
async def _run(body):
|
||||
return await asyncio.wait_for(body(), 10)
|
||||
|
||||
|
||||
def run(body):
|
||||
return asyncio.run(_run(body))
|
||||
|
||||
|
||||
async def _wait_entered(t: FakeTranscriber):
|
||||
while not t.entered.is_set():
|
||||
await asyncio.sleep(0.005)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_finals_queued_while_busy_go_in_one_batch_before_partials():
|
||||
t = FakeTranscriber(hold_first=True)
|
||||
|
||||
async def body():
|
||||
s = SttScheduler(t, max_partial_batch=4)
|
||||
s.start()
|
||||
got = []
|
||||
done = asyncio.Event()
|
||||
|
||||
def cb(name):
|
||||
def _cb(res):
|
||||
got.append((name, res.text))
|
||||
if len(got) == 5:
|
||||
done.set()
|
||||
return _cb
|
||||
|
||||
s.submit_partial(("a", "1"), clip(10), PARTIAL, cb("p-first"))
|
||||
await _wait_entered(t)
|
||||
s.submit_partial(("a", "2"), clip(20), PARTIAL, cb("p2"))
|
||||
s.submit_final(("a", "3"), clip(30), FINAL, cb("f3"))
|
||||
s.submit_final(("a", "4"), clip(40), FINAL, cb("f4"))
|
||||
s.submit_partial(("a", "5"), clip(50), PARTIAL, cb("p5"))
|
||||
t.gate.set()
|
||||
await done.wait()
|
||||
await s.stop()
|
||||
return got
|
||||
|
||||
got = run(body)
|
||||
assert [c[1] for c in t.calls] == [[10], [30, 40], [20, 50]]
|
||||
assert [c[0].beam for c in t.calls] == [1, 2, 1]
|
||||
assert got == [("p-first", "n10"), ("f3", "n30"), ("f4", "n40"), ("p2", "n20"), ("p5", "n50")]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_newer_partial_replaces_the_waiting_one_for_the_same_speaker():
|
||||
t = FakeTranscriber(hold_first=True)
|
||||
|
||||
async def body():
|
||||
s = SttScheduler(t)
|
||||
s.start()
|
||||
got = []
|
||||
done = asyncio.Event()
|
||||
s.submit_final(("a", "x"), clip(5), FINAL, lambda r: None)
|
||||
await _wait_entered(t)
|
||||
s.submit_partial(("a", "1"), clip(10), PARTIAL, lambda r: got.append(r.text))
|
||||
s.submit_partial(("a", "1"), clip(20), PARTIAL, lambda r: (got.append(r.text), done.set()))
|
||||
t.gate.set()
|
||||
await done.wait()
|
||||
await asyncio.sleep(0.05)
|
||||
await s.stop()
|
||||
return got
|
||||
|
||||
assert run(body) == ["n20"]
|
||||
assert [c[1] for c in t.calls] == [[5], [20]]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_dropped_partials_are_never_transcribed():
|
||||
t = FakeTranscriber(hold_first=True)
|
||||
|
||||
async def body():
|
||||
s = SttScheduler(t)
|
||||
s.start()
|
||||
done = asyncio.Event()
|
||||
got = []
|
||||
s.submit_final(("a", "x"), clip(5), FINAL, lambda r: None)
|
||||
await _wait_entered(t)
|
||||
s.submit_partial(("a", "1"), clip(10), PARTIAL, lambda r: got.append(r.text))
|
||||
s.submit_partial(("b", "1"), clip(20), PARTIAL, lambda r: (got.append(r.text), done.set()))
|
||||
s.drop_partials(lambda key: key[0] == "a")
|
||||
t.gate.set()
|
||||
await done.wait()
|
||||
await s.stop()
|
||||
return got
|
||||
|
||||
assert run(body) == ["n20"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_different_decode_settings_are_not_mixed_in_one_batch():
|
||||
t = FakeTranscriber(hold_first=True)
|
||||
other = DecodeOptions(beam=2, language=None, hotwords=("자비스",))
|
||||
|
||||
async def body():
|
||||
s = SttScheduler(t)
|
||||
s.start()
|
||||
n = []
|
||||
done = asyncio.Event()
|
||||
|
||||
def cb(r):
|
||||
n.append(r)
|
||||
if len(n) == 4:
|
||||
done.set()
|
||||
|
||||
s.submit_final(("a", "x"), clip(1), FINAL, cb)
|
||||
await _wait_entered(t)
|
||||
s.submit_final(("a", "1"), clip(10), FINAL, cb)
|
||||
s.submit_final(("b", "1"), clip(20), other, cb)
|
||||
s.submit_final(("a", "2"), clip(30), FINAL, cb)
|
||||
t.gate.set()
|
||||
await done.wait()
|
||||
await s.stop()
|
||||
|
||||
run(body)
|
||||
assert sorted(c[1] for c in t.calls[1:]) == [[10, 30], [20]]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_batches_are_capped():
|
||||
t = FakeTranscriber(hold_first=True)
|
||||
|
||||
async def body():
|
||||
s = SttScheduler(t, max_batch=3)
|
||||
s.start()
|
||||
n = []
|
||||
done = asyncio.Event()
|
||||
|
||||
def cb(r):
|
||||
n.append(r)
|
||||
if len(n) == 8:
|
||||
done.set()
|
||||
|
||||
s.submit_final(("a", "x"), clip(1), FINAL, cb)
|
||||
await _wait_entered(t)
|
||||
for i in range(7):
|
||||
s.submit_final(("a", str(i)), clip(10 + i), FINAL, cb)
|
||||
t.gate.set()
|
||||
await done.wait()
|
||||
await s.stop()
|
||||
|
||||
run(body)
|
||||
assert [len(c[1]) for c in t.calls] == [1, 3, 3, 1]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_failing_batch_reports_the_error_and_the_scheduler_keeps_going():
|
||||
t = FakeTranscriber(fail_on_beam=2)
|
||||
|
||||
async def body():
|
||||
s = SttScheduler(t)
|
||||
s.start()
|
||||
out = []
|
||||
done = asyncio.Event()
|
||||
s.submit_final(("a", "1"), clip(10), FINAL, out.append)
|
||||
s.submit_partial(("a", "2"), clip(20), PARTIAL, lambda r: (out.append(r), done.set()))
|
||||
await done.wait()
|
||||
await s.stop()
|
||||
return out
|
||||
|
||||
out = run(body)
|
||||
assert out[0].text == "" and out[0].error
|
||||
assert out[1].text == "n20" and out[1].error is None
|
||||
assert out[0].stt_ms >= 0
|
||||
|
||||
|
||||
|
||||
class StepTranscriber:
|
||||
"""Every call waits for one ``step()``, so the test decides when each batch finishes."""
|
||||
|
||||
def __init__(self):
|
||||
self.calls: list[list[int]] = []
|
||||
self._steps = threading.Semaphore(0)
|
||||
|
||||
def step(self):
|
||||
self._steps.release()
|
||||
|
||||
def transcribe(self, clips, opts):
|
||||
self.calls.append([len(c) for c in clips])
|
||||
self._steps.acquire(timeout=5)
|
||||
return ["" for _ in clips]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_partial_batches_are_kept_small_so_finals_can_cut_in():
|
||||
t = StepTranscriber()
|
||||
|
||||
async def until(n):
|
||||
while len(t.calls) < n:
|
||||
await asyncio.sleep(0.005)
|
||||
|
||||
async def body():
|
||||
s = SttScheduler(t, max_partial_batch=2)
|
||||
s.start()
|
||||
for i in range(5):
|
||||
s.submit_partial(("a", str(i)), clip(10 + i), PARTIAL, lambda r: None)
|
||||
await until(1) # first partial batch is running
|
||||
s.submit_final(("a", "late"), clip(99), FINAL, lambda r: None)
|
||||
t.step()
|
||||
await until(2)
|
||||
t.step()
|
||||
await until(3)
|
||||
t.step()
|
||||
await until(4)
|
||||
t.step()
|
||||
await asyncio.sleep(0.02)
|
||||
await s.stop()
|
||||
|
||||
run(body)
|
||||
assert t.calls == [[10, 11], [99], [12, 13], [14]]
|
||||
191
tests/test_voice_worker_segmenter.py
Normal file
191
tests/test_voice_worker_segmenter.py
Normal file
@@ -0,0 +1,191 @@
|
||||
"""Voice worker: per-speaker utterance segmentation (start, partial cadence, the four end reasons).
|
||||
|
||||
The fake VAD reads the speech probability straight off the audio: a window of constant amplitude
|
||||
0.9 is "speech at 0.9". That keeps the tests about the segmentation rules, not about Silero.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from voice_worker.segmenter import Final, Partial, Segmenter, SpeechStart
|
||||
from voice_worker.settings import SttSettings
|
||||
|
||||
SR = 16000
|
||||
FRAME_MS = 20
|
||||
|
||||
|
||||
class AmplitudeVad:
|
||||
def prob(self, window: np.ndarray) -> float:
|
||||
return float(np.max(np.abs(window)))
|
||||
|
||||
|
||||
class Feeder:
|
||||
def __init__(self, settings: SttSettings | None = None):
|
||||
self.settings = settings or SttSettings()
|
||||
self.seg = Segmenter(AmplitudeVad(), self.settings)
|
||||
self.now = 100.0
|
||||
self.events: list = []
|
||||
|
||||
def feed(self, level: float, ms: int) -> list:
|
||||
got = []
|
||||
for _ in range(ms // FRAME_MS):
|
||||
self.now += FRAME_MS / 1000
|
||||
got += self.seg.push(np.full(SR * FRAME_MS // 1000, level, dtype=np.float32), self.now)
|
||||
self.events += got
|
||||
return got
|
||||
|
||||
def of(self, cls) -> list:
|
||||
return [e for e in self.events if isinstance(e, cls)]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_silence_produces_nothing():
|
||||
f = Feeder()
|
||||
f.feed(0.0, 3000)
|
||||
assert f.events == []
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_speech_then_silence_ends_with_one_final():
|
||||
f = Feeder()
|
||||
f.feed(0.0, 500)
|
||||
f.feed(0.9, 1000)
|
||||
f.feed(0.0, 500)
|
||||
starts, finals = f.of(SpeechStart), f.of(Final)
|
||||
assert [s.utt for s in starts] == [1]
|
||||
assert len(finals) == 1
|
||||
fin = finals[0]
|
||||
assert fin.utt == 1 and fin.reason == "silence"
|
||||
# Pre-roll (<= 200 ms) + 1 s speech + trailing silence capped at 200 ms.
|
||||
assert 1.0 <= len(fin.audio) / SR <= 1.45
|
||||
assert fin.audio.dtype == np.float32
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_final_waits_for_the_configured_silence():
|
||||
f = Feeder(SttSettings(end_silence_ms=600))
|
||||
f.feed(0.9, 800)
|
||||
f.feed(0.0, 400)
|
||||
assert f.of(Final) == []
|
||||
f.feed(0.0, 300)
|
||||
assert len(f.of(Final)) == 1
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_changed_settings_apply_to_the_running_utterance():
|
||||
f = Feeder(SttSettings(end_silence_ms=1500))
|
||||
f.feed(0.9, 800)
|
||||
f.settings.update({"end_silence_ms": 200})
|
||||
f.feed(0.0, 300)
|
||||
assert [e.reason for e in f.of(Final)] == ["silence"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_short_blip_is_dropped_without_any_message():
|
||||
f = Feeder()
|
||||
f.feed(0.9, 120)
|
||||
f.feed(0.0, 1000)
|
||||
assert f.events == []
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_hysteresis_keeps_speech_going_but_never_starts_it():
|
||||
f = Feeder()
|
||||
f.feed(0.4, 1000)
|
||||
assert f.events == []
|
||||
f.feed(0.9, 400)
|
||||
f.feed(0.4, 1000) # below the start threshold but above the keep threshold
|
||||
assert f.of(Final) == []
|
||||
f.feed(0.0, 400)
|
||||
assert len(f.of(Final)) == 1
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_speaking_end_closes_the_utterance_at_once():
|
||||
f = Feeder()
|
||||
f.feed(0.9, 700)
|
||||
events = f.seg.speaking_end(f.now)
|
||||
assert [type(e) for e in events] == [Final]
|
||||
assert events[0].reason == "speaking_end"
|
||||
assert events[0].decided_at == f.now
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_speaking_end_after_a_blip_sends_nothing():
|
||||
f = Feeder()
|
||||
f.feed(0.9, 100)
|
||||
assert f.seg.speaking_end(f.now) == []
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_audio_that_stops_arriving_ends_as_idle():
|
||||
f = Feeder()
|
||||
f.feed(0.9, 700)
|
||||
assert f.seg.tick(f.now + 0.2) == []
|
||||
events = f.seg.tick(f.now + 0.31)
|
||||
assert [e.reason for e in events] == ["idle"]
|
||||
assert f.seg.tick(f.now + 1.0) == []
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_long_speech_is_cut_at_the_whisper_window_and_continues_as_a_new_utterance():
|
||||
f = Feeder()
|
||||
f.feed(0.9, 29_000)
|
||||
finals = f.of(Final)
|
||||
assert [e.reason for e in finals] == ["max_length"]
|
||||
assert len(finals[0].audio) / SR <= 28.0
|
||||
assert [s.utt for s in f.of(SpeechStart)] == [1, 2]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_partials_follow_the_interval_and_grow():
|
||||
f = Feeder(SttSettings(partial_interval_ms=400))
|
||||
f.feed(0.9, 2000)
|
||||
partials = f.of(Partial)
|
||||
assert 3 <= len(partials) <= 5
|
||||
lengths = [len(p.audio) for p in partials]
|
||||
assert lengths == sorted(lengths) and len(set(lengths)) == len(lengths)
|
||||
assert all(p.utt == 1 for p in partials)
|
||||
f.feed(0.0, 400)
|
||||
n = len(f.of(Partial))
|
||||
f.feed(0.0, 1000)
|
||||
assert len(f.of(Partial)) == n # nothing after the final
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_no_partials_before_speech_start():
|
||||
f = Feeder(SttSettings(partial_interval_ms=100))
|
||||
f.feed(0.9, 200)
|
||||
assert f.of(Partial) == []
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_utterance_numbers_count_up_per_speaker():
|
||||
f = Feeder()
|
||||
for _ in range(3):
|
||||
f.feed(0.9, 500)
|
||||
f.feed(0.0, 400)
|
||||
assert [e.utt for e in f.of(Final)] == [1, 2, 3]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("bad", [{"end_silence_ms": 50}, {"end_silence_ms": 5000}, {"final_beam": 0},
|
||||
{"final_beam": 9}, {"partials": "yes"}, {"hotwords": "자비스"},
|
||||
{"language": 3}, {"partial_interval_ms": 10}, {"nope": 1}])
|
||||
def test_bad_settings_are_refused_and_change_nothing(bad):
|
||||
s = SttSettings()
|
||||
before = s.as_dict()
|
||||
with pytest.raises(ValueError):
|
||||
s.update(bad)
|
||||
assert s.as_dict() == before
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_settings_accept_null_language_and_hotword_lists():
|
||||
s = SttSettings()
|
||||
s.update({"language": None, "hotwords": ["자비스", "Jarvis"], "partials": False})
|
||||
assert s.as_dict()["language"] is None
|
||||
assert s.hotwords == ("자비스", "Jarvis")
|
||||
assert s.partials is False
|
||||
50
tests/test_voice_worker_sentences.py
Normal file
50
tests/test_voice_worker_sentences.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""Voice worker: sentence splitting for streaming TTS, whole and incremental."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from voice_worker.sentences import StreamingSplitter, split_sentences
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_splits_on_terminal_punctuation_only():
|
||||
assert split_sentences("안녕하세요. 오늘은 맑아요! 우산은 필요 없어요?") == [
|
||||
"안녕하세요.", "오늘은 맑아요!", "우산은 필요 없어요?"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_decimals_versions_and_urls_stay_whole():
|
||||
assert split_sentences("기온은 17.5도예요. 자세한 건 example.com 에 있어요.") == [
|
||||
"기온은 17.5도예요.", "자세한 건 example.com 에 있어요."]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_cjk_terminators_split_without_spaces():
|
||||
assert split_sentences("今日は晴れです。明日は雨です。") == ["今日は晴れです。", "明日は雨です。"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_short_fragments_merge_into_a_neighbour():
|
||||
assert split_sentences("네. 알람을 일곱 시에 맞췄어요.") == ["네. 알람을 일곱 시에 맞췄어요."]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_streaming_holds_the_unfinished_tail_until_more_text_arrives():
|
||||
s = StreamingSplitter()
|
||||
assert s.feed("오늘 서울은 맑") == []
|
||||
assert s.feed("아요. 기온은 17") == ["오늘 서울은 맑아요."]
|
||||
assert s.feed(".5도예요. 바람은") == ["기온은 17.5도예요."]
|
||||
assert s.finish() == ["바람은"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_streaming_output_equals_whole_text_split():
|
||||
text = "첫 문장입니다. 두 번째 문장이에요! 세 번째는 질문인가요? 마지막 문장이에요."
|
||||
s = StreamingSplitter()
|
||||
got = []
|
||||
for i in range(0, len(text), 3):
|
||||
got += s.feed(text[i:i + 3])
|
||||
got += s.finish()
|
||||
assert got == split_sentences(text)
|
||||
|
||||
327
tests/test_voice_worker_server.py
Normal file
327
tests/test_voice_worker_server.py
Normal file
@@ -0,0 +1,327 @@
|
||||
"""Voice worker end to end over a real local WebSocket, with fake STT/VAD/TTS engines.
|
||||
|
||||
The fake VAD reads speech probability off the amplitude and the fake transcriber names a clip by its
|
||||
peak level, so each speaker's text tells which speaker's audio reached the model.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from voice_worker.protocol import decode_tts_frame, encode_audio_frame
|
||||
from voice_worker.server import CLOSE_BAD_HELLO, CLOSE_BAD_TOKEN, VoiceWorker
|
||||
|
||||
websockets = pytest.importorskip("websockets")
|
||||
from websockets.asyncio.client import connect # noqa: E402
|
||||
|
||||
TOKEN = "test-token"
|
||||
|
||||
|
||||
class AmplitudeVad:
|
||||
def prob(self, window):
|
||||
return float(np.max(np.abs(window)))
|
||||
|
||||
|
||||
class LevelTranscriber:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def transcribe(self, clips, opts):
|
||||
self.calls.append((opts, len(clips)))
|
||||
return [f"level {np.median(np.abs(c)):.1f}" for c in clips]
|
||||
|
||||
|
||||
class FakeTts:
|
||||
"""10 samples per character; optional delay per sentence; can fail on a marker."""
|
||||
|
||||
def __init__(self, delay: float = 0.0):
|
||||
self.delay = delay
|
||||
self.texts = []
|
||||
|
||||
def synthesize(self, text, rate):
|
||||
if self.delay:
|
||||
time.sleep(self.delay)
|
||||
if "FAIL" in text:
|
||||
raise RuntimeError("engine broke")
|
||||
self.texts.append(text)
|
||||
return np.full(len(text) * 10, 1000, dtype=np.int16)
|
||||
|
||||
|
||||
def pcm(level: float, ms: int, rate=16000, channels=1) -> bytes:
|
||||
n = rate * ms // 1000
|
||||
return (np.full(n * channels, level) * 32767).astype("<i2").tobytes()
|
||||
|
||||
|
||||
async def with_worker(body, transcriber=None, tts=None, **kw):
|
||||
worker = VoiceWorker(TOKEN, transcriber if transcriber is not None else LevelTranscriber(), AmplitudeVad,
|
||||
tts, **kw)
|
||||
server = await worker.serve("127.0.0.1", 0)
|
||||
port = server.sockets[0].getsockname()[1]
|
||||
try:
|
||||
return await asyncio.wait_for(body(f"ws://127.0.0.1:{port}"), 15)
|
||||
finally:
|
||||
server.close()
|
||||
await server.wait_closed()
|
||||
await worker.stop()
|
||||
|
||||
|
||||
async def hello(ws, **extra):
|
||||
await ws.send(json.dumps({"type": "hello", "token": TOKEN, "audio": {"rate": 16000, "channels": 1},
|
||||
"tts": {"rate": 24000, "channels": 2}, **extra}))
|
||||
return json.loads(await ws.recv())
|
||||
|
||||
|
||||
async def recv_until(ws, pred, limit=200):
|
||||
seen = []
|
||||
for _ in range(limit):
|
||||
raw = await ws.recv()
|
||||
msg = raw if isinstance(raw, bytes) else json.loads(raw)
|
||||
seen.append(msg)
|
||||
if pred(msg):
|
||||
return seen
|
||||
raise AssertionError("condition never met")
|
||||
|
||||
|
||||
async def stream(ws, speaker: str, level: float, ms: int, realtime: bool = False):
|
||||
for _ in range(ms // 20):
|
||||
await ws.send(encode_audio_frame(speaker, pcm(level, 20)))
|
||||
if realtime:
|
||||
await asyncio.sleep(0.02)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_wrong_token_is_refused():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await ws.send(json.dumps({"type": "hello", "token": "nope"}))
|
||||
with pytest.raises(websockets.ConnectionClosed) as e:
|
||||
await ws.recv()
|
||||
return e.value.rcvd.code
|
||||
|
||||
assert asyncio.run(with_worker(body)) == CLOSE_BAD_TOKEN
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_silent_client_is_dropped_after_the_hello_timeout():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
with pytest.raises(websockets.ConnectionClosed) as e:
|
||||
await ws.recv()
|
||||
return e.value.rcvd.code
|
||||
|
||||
assert asyncio.run(with_worker(body, hello_timeout_s=0.2)) == CLOSE_BAD_HELLO
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_non_loopback_addresses_are_refused():
|
||||
worker = VoiceWorker(TOKEN, LevelTranscriber(), AmplitudeVad)
|
||||
with pytest.raises(ValueError):
|
||||
asyncio.run(worker.serve("0.0.0.0", 0))
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_hello_ok_reports_readiness_and_effective_settings():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
return await hello(ws, stt={"hotwords": ["자비스"], "end_silence_ms": 250})
|
||||
|
||||
ok = asyncio.run(with_worker(body))
|
||||
assert ok["type"] == "hello_ok"
|
||||
assert ok["stt_ready"] is True and ok["tts_ready"] is False
|
||||
assert ok["stt"]["hotwords"] == ["자비스"] and ok["stt"]["end_silence_ms"] == 250
|
||||
assert ok["tts"] == {"rate": 24000, "channels": 2}
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_two_speakers_each_get_their_own_final():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws, stt={"partials": False})
|
||||
await stream(ws, "alice", 0.9, 600)
|
||||
await stream(ws, "bob", 0.6, 600)
|
||||
await ws.send(json.dumps({"type": "speaking_end", "speaker": "alice"}))
|
||||
await ws.send(json.dumps({"type": "speaking_end", "speaker": "bob"}))
|
||||
seen = await recv_until(ws, lambda m: m.get("type") == "final" and m["speaker"] == "bob")
|
||||
if not any(m.get("type") == "final" and m["speaker"] == "alice" for m in seen):
|
||||
seen += await recv_until(ws, lambda m: m.get("type") == "final")
|
||||
return seen
|
||||
|
||||
seen = asyncio.run(with_worker(body))
|
||||
finals = {m["speaker"]: m for m in seen if m.get("type") == "final"}
|
||||
assert finals["alice"]["text"] == "level 0.9" and finals["bob"]["text"] == "level 0.6"
|
||||
assert {m["end_reason"] for m in finals.values()} == {"speaking_end"}
|
||||
assert all(m["utt"] == 1 and 500 <= m["audio_ms"] <= 900 for m in finals.values())
|
||||
starts = [m for m in seen if m.get("type") == "speech_start"]
|
||||
assert sorted(m["speaker"] for m in starts) == ["alice", "bob"]
|
||||
assert not any(m.get("type") == "partial" for m in seen)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_audio_that_stops_without_speaking_end_still_ends():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws, stt={"partials": False, "end_silence_ms": 200})
|
||||
await stream(ws, "alice", 0.9, 500)
|
||||
return await recv_until(ws, lambda m: m.get("type") == "final")
|
||||
|
||||
final = asyncio.run(with_worker(body))[-1]
|
||||
assert final["end_reason"] == "idle" and final["text"] == "level 0.9"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_partials_flow_while_speaking_and_stop_when_turned_off():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws, stt={"partial_interval_ms": 100})
|
||||
await stream(ws, "alice", 0.9, 800, realtime=True)
|
||||
first = await recv_until(ws, lambda m: m.get("type") == "partial")
|
||||
await ws.send(json.dumps({"type": "config", "partials": False}))
|
||||
# Drain whatever was already on its way before the setting changed.
|
||||
while True:
|
||||
try:
|
||||
await asyncio.wait_for(ws.recv(), 0.3)
|
||||
except asyncio.TimeoutError:
|
||||
break
|
||||
await stream(ws, "alice", 0.9, 600, realtime=True)
|
||||
await ws.send(json.dumps({"type": "speaking_end", "speaker": "alice"}))
|
||||
rest = await recv_until(ws, lambda m: m.get("type") == "final")
|
||||
return first, rest
|
||||
|
||||
first, rest = asyncio.run(with_worker(body))
|
||||
assert first[-1]["text"] == "level 0.9" and first[-1]["utt"] == 1
|
||||
assert not any(m.get("type") == "partial" for m in rest)
|
||||
assert rest[-1]["type"] == "final"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_tts_streams_one_chunk_per_sentence_in_the_session_format():
|
||||
tts = FakeTts()
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await ws.send(json.dumps({"type": "tts", "id": "r1", "text": "오늘은 맑아요. 기온은 17.5도예요."}))
|
||||
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
|
||||
|
||||
seen = asyncio.run(with_worker(body, tts=tts))
|
||||
chunks = [decode_tts_frame(m) for m in seen if isinstance(m, bytes)]
|
||||
assert [(c[0], c[1]) for c in chunks] == [("r1", 0), ("r1", 1)]
|
||||
assert tts.texts == ["오늘은 맑아요.", "기온은 17.5도예요."]
|
||||
# Stereo: every mono sample twice, 16-bit.
|
||||
assert len(chunks[0][2]) == len("오늘은 맑아요.") * 10 * 2 * 2
|
||||
assert seen[-1] == {"type": "tts_done", "id": "r1", "chunks": 2, "cancelled": False, "failed": False}
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_tts_text_can_arrive_in_pieces():
|
||||
tts = FakeTts()
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
for piece, more in [("첫 번째 문장", True), ("이에요. 두 번", True), ("째 문장이에요.", False)]:
|
||||
await ws.send(json.dumps({"type": "tts", "id": "s", "text": piece, "more": more}))
|
||||
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
|
||||
|
||||
seen = asyncio.run(with_worker(body, tts=tts))
|
||||
assert tts.texts == ["첫 번째 문장이에요.", "두 번째 문장이에요."]
|
||||
assert seen[-1]["chunks"] == 2
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_cancel_stops_the_remaining_sentences():
|
||||
tts = FakeTts(delay=0.15)
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
text = " ".join(f"이것은 {i}번째 문장입니다." for i in range(6))
|
||||
await ws.send(json.dumps({"type": "tts", "id": "c", "text": text}))
|
||||
first = await recv_until(ws, lambda m: isinstance(m, bytes))
|
||||
await ws.send(json.dumps({"type": "tts_cancel", "id": "c"}))
|
||||
return first + await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
|
||||
|
||||
seen = asyncio.run(with_worker(body, tts=tts))
|
||||
done = seen[-1]
|
||||
assert done["cancelled"] is True and done["chunks"] < 6
|
||||
assert sum(isinstance(m, bytes) for m in seen) == done["chunks"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_failing_sentence_ends_the_request_as_failed():
|
||||
tts = FakeTts()
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await ws.send(json.dumps({"type": "tts", "id": "f", "text": "첫 문장입니다. FAIL 문장입니다. 셋째 문장입니다."}))
|
||||
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
|
||||
|
||||
seen = asyncio.run(with_worker(body, tts=tts))
|
||||
errors = [m for m in seen if isinstance(m, dict) and m.get("type") == "error"]
|
||||
assert errors and errors[0]["code"] == "tts_failed" and errors[0]["id"] == "f"
|
||||
assert seen[-1]["failed"] is True and seen[-1]["chunks"] == 1
|
||||
assert tts.texts == ["첫 문장입니다."]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_tts_without_an_engine_reports_unavailable():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await ws.send(json.dumps({"type": "tts", "id": "x", "text": "안녕하세요."}))
|
||||
return json.loads(await ws.recv())
|
||||
|
||||
msg = asyncio.run(with_worker(body))
|
||||
assert msg["type"] == "error" and msg["code"] == "tts_unavailable" and msg["id"] == "x"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_bad_messages_get_an_error_and_the_connection_stays_open():
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
out = []
|
||||
for raw in ["{not json", json.dumps({"type": "dance"}), json.dumps({"type": "config", "final_beam": 99})]:
|
||||
await ws.send(raw)
|
||||
out.append(json.loads(await ws.recv()))
|
||||
await ws.send(b"\x01\x00")
|
||||
out.append(json.loads(await ws.recv()))
|
||||
await ws.send(json.dumps({"type": "config", "final_beam": 3}))
|
||||
await ws.send(json.dumps({"type": "dance"}))
|
||||
out.append(json.loads(await ws.recv()))
|
||||
return out
|
||||
|
||||
out = asyncio.run(with_worker(body))
|
||||
assert [m["code"] for m in out] == ["bad_message", "bad_message", "bad_message", "bad_frame", "bad_message"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_48k_stereo_input_reaches_the_model_at_16k():
|
||||
t = LevelTranscriber()
|
||||
lengths = []
|
||||
orig = t.transcribe
|
||||
|
||||
def spy(clips, opts):
|
||||
lengths.extend(len(c) for c in clips)
|
||||
return orig(clips, opts)
|
||||
|
||||
t.transcribe = spy
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await ws.send(json.dumps({"type": "hello", "token": TOKEN, "stt": {"partials": False}}))
|
||||
await ws.recv()
|
||||
for _ in range(30): # 600 ms
|
||||
await ws.send(encode_audio_frame("a", pcm(0.9, 20, rate=48000, channels=2)))
|
||||
await ws.send(json.dumps({"type": "speaking_end", "speaker": "a"}))
|
||||
return await recv_until(ws, lambda m: m.get("type") == "final")
|
||||
|
||||
final = asyncio.run(with_worker(body, transcriber=t))[-1]
|
||||
assert final["text"] == "level 0.9"
|
||||
assert 0.5 * 16000 <= lengths[0] <= 0.9 * 16000
|
||||
53
tests/test_voice_worker_vad.py
Normal file
53
tests/test_voice_worker_vad.py
Normal file
@@ -0,0 +1,53 @@
|
||||
"""Voice worker: the streaming Silero VAD (real ONNX model bundled with faster-whisper, CPU only)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import wave
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("onnxruntime")
|
||||
pytest.importorskip("faster_whisper")
|
||||
|
||||
from voice_worker.segmenter import WINDOW # noqa: E402
|
||||
from voice_worker.vad import SileroVad # noqa: E402
|
||||
|
||||
SAMPLE = Path(__file__).resolve().parent.parent / "scripts" / "m0" / "assets" / "ko_samples" / "ko_01.wav"
|
||||
|
||||
|
||||
def _speech() -> np.ndarray:
|
||||
with wave.open(str(SAMPLE), "rb") as w:
|
||||
return np.frombuffer(w.readframes(w.getnframes()), dtype="<i2").astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
def _probs(stream, audio):
|
||||
n = len(audio) // WINDOW * WINDOW
|
||||
return np.array([stream.prob(audio[i:i + WINDOW]) for i in range(0, n, WINDOW)])
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def vad():
|
||||
return SileroVad()
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_speech_is_detected_and_silence_is_not(vad):
|
||||
speech = _probs(vad.stream(), _speech())
|
||||
silence = _probs(vad.stream(), np.zeros(16000, dtype=np.float32))
|
||||
assert (speech >= 0.5).mean() > 0.4
|
||||
assert silence.max() < 0.2
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_each_speaker_stream_keeps_its_own_state(vad):
|
||||
audio = _speech()
|
||||
alone = _probs(vad.stream(), audio)
|
||||
a, b = vad.stream(), vad.stream()
|
||||
n = len(audio) // WINDOW * WINDOW
|
||||
interleaved = []
|
||||
for i in range(0, n, WINDOW):
|
||||
interleaved.append(a.prob(audio[i:i + WINDOW]))
|
||||
b.prob(np.zeros(WINDOW, dtype=np.float32)) # another speaker in silence between every window
|
||||
assert np.allclose(alone, interleaved, atol=1e-5)
|
||||
Reference in New Issue
Block a user