feat(voice_worker): M1 multi-speaker streaming STT and sentence TTS over a local WebSocket

- One shared faster-whisper model for every speaker, batched; finals go before partials and
  partial batches are capped at two so a final never waits behind a ten-speaker pass.
- Per-speaker 48k stereo to 16k mono conversion, streaming Silero VAD, utterance ends on VAD
  silence, Discord speaking_end, missing audio, or the 28 s Whisper window.
- MeloTTS in its own venv through a child process; sentences stream out one chunk each, with
  incremental text, cancel and failure reporting.
- Loopback-only server with a token from the environment; logs carry no transcript text.
- Spec in src/voice_worker/voice_worker.spec.md; unit tests with fake engines and an opt-in
  real-GPU test that streams ten speakers at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-11 10:00:06 +09:00
parent b4926d5e19
commit a7685e9078
26 changed files with 2660 additions and 1 deletions

View File

@@ -28,6 +28,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
| `src/jarvis/memory/graph.spec.md` | Node graph memory (v2), self-organising tree, UI explorer | Dynamic structure; access-aware; auto-split/merge (future) |
| `src/jarvis/memory/summariser.spec.md` | Diary summariser prompt contract, hygiene rules (deflection, attribution, topic separation), post-process scrub, and bulk-sweep clean button | Two-layer defence: prompt + deterministic scrub; corrupted summaries poison every downstream consumer |
| `src/jarvis/memory/recall_gate.spec.md` | Deterministic skip-enrichment heuristic when the hot window covers a follow-up | Fail-open; language-agnostic via `\w{3,}` + `re.UNICODE`; planner intent always wins |
| `src/voice_worker/voice_worker.spec.md` | v2 voice worker: multi-speaker streaming STT (shared Whisper, batched) and sentence TTS over a local WebSocket | Loopback only with an app token; no transcript text in logs; finals before partials; language-agnostic |
The LLM contexts graph at `docs/llm_contexts.md` maps every LLM call in the app (model, gating, inputs, outputs, limits, flow). Keep it up-to-date at all times: any change that adds, removes, or alters an LLM context (model resolution, timeout, cap, prompt source, gating flag, data-flow edge) must update `docs/llm_contexts.md` in the same PR.

View File

@@ -5,7 +5,7 @@
| 단계 | 내용 | 완료 기준 |
|---|---|---|
| M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. 가상 모니터 확인을 뺀 모든 항목 측정 완료(2026-11-10). 가상 모니터 확인은 M6에서 합니다. 2026-11-10에 추가한 `desktop` 단계(agent-desktop)는 측정을 마쳤고, 자동 성능 조절용 `gamemode` 단계는 세 번 쟀고, `gamemode_stt`(LLM과 STT를 함께 켬, STT GPU·CPU)와 `gamemode_live`(실제 게임을 켬) 단계도 측정을 마쳤습니다(2026-11-10). 결과로 게임 중 최소 배치는 전문가 2GB로 정했고, 평소 배치에 STT 자리를 남기는 것과 게임 중 LLM이 답하는 동안 부분 전사 간격을 늘리는 것(게임이 없을 때는 그대로)도 정했습니다(사용자 결정, 2026-11-10). `gamemode_tune` 단계도 측정을 마쳤고(2026-11-11), 게임 중 LLM이 답하는 동안에는 부분 전사를 멈추기로 정했습니다. GPU 95% 제한은 측정에서 사용률을 낮추지 못해 기본값을 끔으로 바꿨고(사용자 결정, 2026-11-11), `video_drops` 단계(Jarvis를 돌리는 동안 유튜브 영상의 끊긴 프레임, 직접 실행)도 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 영상이 끊기지 않았고 제한이 끊김을 줄이지 못해 설정을 뺐습니다. 대신 게임 화면의 프레임을 직접 재는 `game_frames` 단계(게임을 켠 채로 Jarvis를 끈 때·쉴 때·바쁠 때와 우선순위를 낮춘 양보 상태, 직접 실행, 관리자 권한)를 넣었고 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 게임의 평균 fps와 1% low가 5% 넘게 떨어지지 않아 양보는 넣지 않기로 했습니다. 남은 M0 측정은 가상 모니터 확인(M6)입니다. 측정 키트: [scripts/m0/](../../scripts/m0/README.md) |
| M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과 |
| M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과. 구현과 테스트 완료(2026-11-11): `src/voice_worker/`, 스펙은 [voice_worker.spec.md](../../src/voice_worker/voice_worker.spec.md). 이 서버의 RTX 5050에서 10명이 겹쳐 말하는 실시간 입력(48kHz 스테레오, M0 한국어 샘플 8개를 10명에게 나눠 줌, 절반은 디스코드 speaking 종료, 절반은 무음으로 끝남)을 넣어 10명 모두 최종 전사를 받았고 문자 오류율은 평균 4%, 최대 19%("일곱 시"를 "7시"로 적은 것), 말 끝 판정부터 최종 전사 전송까지 중앙값 0.64초, 최대 1.56초였습니다. TTS는 첫 문장 소리가 0.2초에 나왔고, 그 소리를 다시 넣으면 같은 문장으로 받아 적혔습니다. 샘플이 MeloTTS 합성 음성이라 실제 사람 녹음과 게임PC 수치는 아직 없습니다 |
| M2 | 디스코드 음성 코어 이전 | v1 봇 코드를 Electron 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기) |
| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과 |
| M4 | 왕복 완성과 지연 튜닝 | 호출어·호출 방식·지정한 사람 설정, 동시 대화 응답 정책 4가지(기본 "처음 대화만")와 정책별 끼어들기 규칙. 말 끝부터 첫 소리까지 실측 1.5초 이내, 10명 채널 부하 테스트 |

View File

@@ -21,6 +21,7 @@
- **종료 판정 단축**: 고정 0.8초 침묵 대기 대신 디스코드 speaking 종료 이벤트와 VAD를 같이 써서 250~400ms 안에 확정합니다(설정에서 조절).
- **2단 디코딩**: 부분 전사는 beam 1(빠름), 최종 전사는 beam 2~3(정확도)입니다.
- **게임 중 LLM이 답하는 동안**: STT와 LLM은 같은 GPU를 씁니다. 10명이 쉬지 않고 말하며 부분 전사를 계속 돌리면 LLM 출력이 1/2.5~1/3로 줄었습니다(M0 `gamemode_stt`). 그래서 게임 중에 LLM이 답을 만드는 동안에는 부분 전사를 멈추고(그동안 실시간 자막이 바뀌지 않음), 말이 끝난 뒤의 최종 전사는 기다리지 않고 바로 처리합니다. 게임이 없을 때는 LLM이 답하는 동안에도 부분 전사를 그대로 합니다(사용자 결정, 2026-11-10). 최종 전사가 부분 전사보다 먼저입니다. 멈추기로 한 것은 M0 `gamemode_tune` 측정 결과입니다(2026-11-11): 2초 간격은 37.5 tok/s, 음성 첫 문장 0.89초에 최종 전사 대기가 최대 1.2초였고, 멈추면 64.9 tok/s, 0.55초였습니다. 부분 전사를 멈췄을 때 beam 2~3 최종 전사와 긴 말의 실제 지연은 아직 재지 않았습니다([pc-features.md](pc-features.md#자동-조절-측정-m0-gamemode_tune-2026-11-11)).
- **부분 전사는 2명씩만 묶음(M1 실측)**: Whisper 인코더는 말 길이와 상관없이 늘 30초 창을 계산해서, 묶어도 비용이 사람 수에 비례합니다(RTX 5050에서 1명 0.2초, 10명 2.1초). 10명분 부분 전사를 한 번에 돌리면 그동안 들어온 최종 전사가 2초 넘게 기다려서, 부분 전사는 한 번에 2명까지만 돌리고 그 사이에 최종 전사가 먼저 들어가게 했습니다. 여러 명이 함께 말하면 자막은 늦어지고 사람마다 가장 최근 소리만 전사합니다. 인코더 창을 줄이면 4~7배 빨라졌지만 같은 문장을 되풀이하는 오류가 생겨 쓰지 않습니다.
- **호출어 힌트**: 등록한 호출어를 STT에 힌트로 넘겨 잘못 알아듣는 일을 줄입니다.
- **RealtimeSTT 사용 범위**: RealtimeSTT는 화자 하나에 인스턴스 하나 구조라 10명이면 모델을 10번 올립니다. 그래서 설계(부분/최종 전사, VAD 결합)만 참고하고, **모델 1개를 10명이 공유하는 배치 방식**으로 직접 구현합니다.
- **선택지**: 더 낮은 지연이 필요하면 sherpa-onnx 스트리밍 모델을 "초고속" 옵션으로 검토합니다(한국어 정확도 확인 후).

View File

@@ -0,0 +1,4 @@
"""Jarvis v2 voice worker: multi-speaker streaming STT and sentence TTS over a local WebSocket.
See ``voice_worker.spec.md`` next to this file.
"""

View File

@@ -0,0 +1,66 @@
"""``JARVIS_VOICE_TOKEN=... python -m voice_worker`` (see voice_worker.spec.md)."""
from __future__ import annotations
import argparse
import asyncio
import os
import sys
from jarvis.debug import debug_log
def main() -> int:
ap = argparse.ArgumentParser(prog="voice_worker")
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, default=8771)
ap.add_argument("--stt-model", default="large-v3-turbo")
ap.add_argument("--stt-device", default="cuda")
ap.add_argument("--stt-compute", default="int8_float16")
ap.add_argument("--melo-python", default=None, help="python of the MeloTTS venv; no TTS without it")
ap.add_argument("--melo-device", default="cuda")
ap.add_argument("--melo-speed", type=float, default=1.5)
ap.add_argument("--melo-language", default="KR")
args = ap.parse_args()
token = os.environ.get("JARVIS_VOICE_TOKEN", "")
if not token:
print("JARVIS_VOICE_TOKEN is not set", file=sys.stderr)
return 2
from .server import VoiceWorker, is_loopback
from .stt_whisper import WhisperTranscriber
from .vad import SileroVad
if not is_loopback(args.host):
print("the voice worker only listens on a loopback address", file=sys.stderr)
return 2
transcriber = WhisperTranscriber(args.stt_model, args.stt_device, args.stt_compute)
transcriber.warm_up()
vad = SileroVad()
tts = None
if args.melo_python:
from .tts_melo import MeloEngine
tts = MeloEngine(args.melo_python, args.melo_device, args.melo_speed, args.melo_language)
async def run() -> None:
worker = VoiceWorker(token, transcriber, vad.stream, tts)
server = await worker.serve(args.host, args.port)
debug_log(f"voice worker listening on {args.host}:{args.port} tts={'on' if tts else 'off'}",
"voice_worker")
print(f"READY {args.host}:{args.port}", flush=True)
await server.serve_forever()
try:
asyncio.run(run())
except KeyboardInterrupt:
pass
finally:
if tts:
tts.close()
return 0
if __name__ == "__main__":
sys.exit(main())

58
src/voice_worker/audio.py Normal file
View File

@@ -0,0 +1,58 @@
"""Incoming PCM to 16 kHz mono float32, and clip normalisation before transcription."""
from __future__ import annotations
import numpy as np
TARGET_RATE = 16000
_TAPS = 97 # windowed-sinc low-pass for the 48 kHz -> 16 kHz decimation
_CUTOFF_HZ = 7200 # below the new 8 kHz Nyquist
def _lowpass(rate: int) -> np.ndarray:
n = np.arange(_TAPS) - (_TAPS - 1) / 2
fc = _CUTOFF_HZ / rate
h = 2 * fc * np.sinc(2 * fc * n) * np.blackman(_TAPS)
return (h / h.sum()).astype(np.float64)
class InputConverter:
"""Streaming converter for one speaker. Keeps filter history and odd bytes between frames."""
def __init__(self, rate: int, channels: int):
if rate not in (16000, 48000):
raise ValueError("audio rate must be 16000 or 48000")
if channels not in (1, 2):
raise ValueError("audio channels must be 1 or 2")
self.channels = channels
self.factor = rate // TARGET_RATE
self._carry = b""
if self.factor > 1:
self._h = _lowpass(rate)
self._hist = np.zeros(_TAPS - 1, dtype=np.float64)
self._phase = 0 # input samples to skip before the next kept output
def push(self, pcm: bytes) -> np.ndarray:
data = self._carry + pcm
frame_bytes = 2 * self.channels
usable = len(data) - len(data) % frame_bytes
self._carry = data[usable:]
x = np.frombuffer(data[:usable], dtype="<i2").astype(np.float64) / 32768.0
if self.channels == 2:
x = x.reshape(-1, 2).mean(axis=1)
if self.factor == 1:
return x.astype(np.float32)
buf = np.concatenate([self._hist, x])
self._hist = buf[-(_TAPS - 1):]
y = np.convolve(buf, self._h, mode="valid") # one output per new input sample
out = y[self._phase::self.factor]
self._phase = (self._phase - len(y)) % self.factor
return out.astype(np.float32)
def peak_normalise(x: np.ndarray, target: float = 0.9, max_gain: float = 10.0) -> np.ndarray:
peak = float(np.max(np.abs(x))) if len(x) else 0.0
if peak <= 0:
return x.astype(np.float32)
gain = min(target / peak, max_gain)
return (x * gain).astype(np.float32)

View File

@@ -0,0 +1,103 @@
"""MeloTTS child process for the voice worker. Runs in its own Python 3.11 venv (transformers 4.27).
Stand-alone on purpose: it must not import the worker package, which lives in another venv.
stdin: one JSON request per line: {"text": "...", "rate": 48000}
stdout: frames of u32 LE header length | header JSON | payload. The first frame is
{"ready": true, "rate": <native rate>}; each reply is {"ok": true, "samples": N} followed by N
int16 LE mono samples at the requested rate, or {"ok": false, "error": "..."} with no payload.
Library output that would land on stdout is sent to stderr so it cannot corrupt the frames.
"""
from __future__ import annotations
import argparse
import importlib
import json
import os
import struct
import sys
import types
def korean_only_patches() -> None:
"""MeloTTS imports its Japanese front end (module-level MeCab.Tagger()) even for Korean, and on
Windows g2pkk asks for ``eunjeon`` instead of python-mecab-ko. Same fix as the M0 kit."""
try:
import MeCab # noqa: F401
MeCab.Tagger # noqa: B018
except Exception:
stub = types.ModuleType("MeCab")
class Tagger:
def __init__(self, *a, **k):
pass
def parse(self, text):
raise RuntimeError("Japanese MeCab is not installed")
stub.Tagger = Tagger
sys.modules["MeCab"] = stub
try:
import g2pkk.g2pkk as g2p_mod
except Exception:
return
if g2p_mod.platform.system() == "Windows":
g2p_mod.G2p.check_mecab = lambda self: None
g2p_mod.G2p.get_mecab = lambda self: importlib.import_module("mecab").MeCab()
def main() -> int:
# Python puts this file's folder first on sys.path; drop it so the worker's own modules there
# never shadow the MeloTTS packages.
here = os.path.dirname(os.path.abspath(__file__))
sys.path[:] = [p for p in sys.path if os.path.abspath(p or ".") != here]
ap = argparse.ArgumentParser()
ap.add_argument("--language", default="KR")
ap.add_argument("--device", default="cuda")
ap.add_argument("--speed", type=float, default=1.5)
args = ap.parse_args()
out = os.fdopen(os.dup(sys.stdout.fileno()), "wb")
os.dup2(sys.stderr.fileno(), sys.stdout.fileno())
sys.stdout = sys.stderr
def reply(header: dict, payload: bytes = b"") -> None:
raw = json.dumps(header).encode()
out.write(struct.pack("<I", len(raw)) + raw + payload)
out.flush()
try:
korean_only_patches()
import numpy as np
from melo.api import TTS
from scipy.signal import resample_poly
model = TTS(language=args.language, device=args.device)
spk_map = model.hps.data.spk2id
speaker = spk_map[args.language] if args.language in spk_map else spk_map[list(spk_map.keys())[0]]
native = int(model.hps.data.sampling_rate)
model.tts_to_file("워밍업입니다.", speaker, None, speed=args.speed, quiet=True)
except Exception as e:
reply({"ready": False, "error": f"{type(e).__name__}: {e}"})
return 1
reply({"ready": True, "rate": native})
for line in sys.stdin:
try:
req = json.loads(line)
rate = int(req["rate"])
audio = model.tts_to_file(str(req["text"]), speaker, None, speed=args.speed, quiet=True)
audio = np.asarray(audio, dtype=np.float64)
if rate != native:
from math import gcd
g = gcd(rate, native)
audio = resample_poly(audio, rate // g, native // g)
pcm = (np.clip(audio, -1.0, 1.0) * 32767).astype("<i2")
reply({"ok": True, "samples": int(len(pcm))}, pcm.tobytes())
except Exception as e:
reply({"ok": False, "error": f"{type(e).__name__}: {e}"})
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,56 @@
"""Binary frame layout shared with the Electron voice core (see voice_worker.spec.md)."""
from __future__ import annotations
import struct
KIND_AUDIO = 0x01
KIND_TTS = 0x02
class ProtocolError(ValueError):
"""A frame or message that does not follow the protocol."""
def _encode_id(value: str) -> bytes:
raw = value.encode("utf-8")
if not 0 < len(raw) < 256:
raise ProtocolError("id must be 1-255 bytes of UTF-8")
return bytes([len(raw)]) + raw
def _decode_id(frame: bytes, kind: int) -> tuple[str, int]:
if len(frame) < 2 or frame[0] != kind:
raise ProtocolError("unexpected frame kind")
n = frame[1]
end = 2 + n
if n == 0 or len(frame) < end:
raise ProtocolError("truncated id")
try:
return frame[2:end].decode("utf-8"), end
except UnicodeDecodeError as e:
raise ProtocolError("id is not UTF-8") from e
def encode_audio_frame(speaker: str, pcm: bytes) -> bytes:
return bytes([KIND_AUDIO]) + _encode_id(speaker) + pcm
def decode_audio_frame(frame: bytes) -> tuple[str, bytes]:
speaker, end = _decode_id(frame, KIND_AUDIO)
pcm = frame[end:]
if not pcm or len(pcm) % 2:
raise ProtocolError("audio payload must be a non-empty run of 16-bit samples")
return speaker, pcm
def encode_tts_frame(request_id: str, seq: int, pcm: bytes) -> bytes:
return bytes([KIND_TTS]) + _encode_id(request_id) + struct.pack("<I", seq) + pcm
def decode_tts_frame(frame: bytes) -> tuple[str, int, bytes]:
request_id, end = _decode_id(frame, KIND_TTS)
if len(frame) < end + 4:
raise ProtocolError("truncated sequence number")
(seq,) = struct.unpack_from("<I", frame, end)
return request_id, seq, frame[end + 4:]

View File

@@ -0,0 +1,12 @@
# Voice worker venv (Python 3.12). Same STT pins the M0 kit measured on the game PC.
faster-whisper==1.2.1
ctranslate2==4.8.1
nvidia-cublas-cu12==12.9.2.10
nvidia-cudnn-cu12==9.24.0.43
onnxruntime>=1.17
websockets>=13
numpy
# jarvis.debug (debug_log) reads the jarvis config, which needs python-dotenv.
python-dotenv==1.0.1
# MeloTTS runs in its own Python 3.11 venv (see the "melo" entry in scripts/m0/m0lib/envs.py) and is
# reached through melo_child.py; pass that venv's python with --melo-python.

View File

@@ -0,0 +1,128 @@
"""One shared STT model for every speaker: finals first, then partials, each batched.
All queue state lives on the asyncio loop; the model runs on one executor thread.
"""
from __future__ import annotations
import asyncio
import time
from collections import OrderedDict
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from typing import Callable, Hashable, Protocol
import numpy as np
from jarvis.debug import debug_log
@dataclass(frozen=True)
class DecodeOptions:
beam: int
language: str | None
hotwords: tuple[str, ...]
@dataclass
class SttResult:
text: str
stt_ms: int
error: str | None = None
class Transcriber(Protocol):
def transcribe(self, clips: list[np.ndarray], opts: DecodeOptions) -> list[str]: ...
Callback = Callable[[SttResult], None]
@dataclass
class _Job:
key: Hashable
audio: np.ndarray
opts: DecodeOptions
callback: Callback
class SttScheduler:
def __init__(self, transcriber: Transcriber, max_batch: int = 16, max_partial_batch: int = 2):
self.transcriber = transcriber
self.max_batch = max_batch
# Whisper's encoder always runs on a 30 s window, so its cost grows with every clip in a batch
# and batching saves little. A small partial batch bounds how long a final can wait behind one.
self.max_partial_batch = max_partial_batch
self._finals: list[_Job] = []
self._partials: "OrderedDict[Hashable, _Job]" = OrderedDict()
self._wake = asyncio.Event()
self._executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="stt")
self._task: asyncio.Task | None = None
def start(self) -> None:
self._task = asyncio.get_running_loop().create_task(self._run())
async def stop(self) -> None:
if self._task:
self._task.cancel()
try:
await self._task
except asyncio.CancelledError:
pass
self._executor.shutdown(wait=False, cancel_futures=True)
def submit_final(self, key: Hashable, audio: np.ndarray, opts: DecodeOptions, callback: Callback) -> None:
self._finals.append(_Job(key, audio, opts, callback))
self._wake.set()
def submit_partial(self, key: Hashable, audio: np.ndarray, opts: DecodeOptions, callback: Callback) -> None:
self._partials.pop(key, None) # a newer partial replaces the waiting one
self._partials[key] = _Job(key, audio, opts, callback)
self._wake.set()
def drop_partials(self, predicate: Callable[[Hashable], bool]) -> None:
for key in [k for k in self._partials if predicate(k)]:
del self._partials[key]
def _take(self) -> tuple[str, list[_Job]]:
if self._finals:
kind, pool = "final", self._finals
elif self._partials:
kind, pool = "partial", list(self._partials.values())
else:
return "", []
opts = pool[0].opts
cap = self.max_batch if kind == "final" else self.max_partial_batch
batch = [j for j in pool if j.opts == opts][:cap]
if kind == "final":
taken = {id(j) for j in batch}
self._finals = [j for j in self._finals if id(j) not in taken]
else:
for j in batch:
del self._partials[j.key]
return kind, batch
async def _run(self) -> None:
loop = asyncio.get_running_loop()
while True:
kind, batch = self._take()
if not batch:
self._wake.clear()
await self._wake.wait()
continue
clips = [j.audio for j in batch]
t0 = time.perf_counter()
try:
texts = await loop.run_in_executor(self._executor, self.transcriber.transcribe, clips,
batch[0].opts)
error = None
except Exception as e: # the model failing must not stop every speaker's STT
texts, error = [""] * len(batch), f"{type(e).__name__}: {e}"
ms = int((time.perf_counter() - t0) * 1000)
debug_log(f"stt batch kind={kind} size={len(batch)} beam={batch[0].opts.beam} ms={ms}"
+ (f" error={error}" if error else ""), "voice_worker")
for job, text in zip(batch, texts):
try:
job.callback(SttResult(text, ms, error))
except Exception as e: # pragma: no cover - a broken session must not stop the loop
debug_log(f"stt callback failed: {type(e).__name__}: {e}", "voice_worker")

View File

@@ -0,0 +1,138 @@
"""Per-speaker utterance segmentation on 16 kHz mono audio (rules in voice_worker.spec.md)."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Protocol
import numpy as np
from .settings import SttSettings
SR = 16000
WINDOW = 512 # Silero window at 16 kHz
WINDOW_MS = WINDOW * 1000 / SR # 32 ms
START_PROB = 0.5
KEEP_PROB = 0.35
PRE_ROLL_MS = 200
MIN_SPEECH_MS = 250
TAIL_KEEP_MS = 200
MAX_UTTERANCE_MS = 28000
class VadStream(Protocol):
def prob(self, window: np.ndarray) -> float: ...
@dataclass
class SpeechStart:
utt: int
@dataclass
class Partial:
utt: int
audio: np.ndarray
@dataclass
class Final:
utt: int
audio: np.ndarray
reason: str # silence | speaking_end | idle | max_length
decided_at: float
def _windows(ms: float) -> int:
return int(ms // WINDOW_MS)
class Segmenter:
def __init__(self, vad: VadStream, settings: SttSettings):
self.vad = vad
self.settings = settings
self._pending = np.zeros(0, dtype=np.float32)
self._pre: list[np.ndarray] = []
self._utt_count = 0
self._last_audio_at: float | None = None
self._reset()
def _reset(self) -> None:
self._active = False
self._win: list[np.ndarray] = []
self._speech = 0
self._trailing = 0
self._started = False
self._last_partial_at = 0.0
self._last_partial_len = 0
@property
def open_utt(self) -> int | None:
"""The utterance that sent speech_start and has not ended yet."""
return self._utt_count if self._active and self._started else None
def push(self, samples: np.ndarray, now: float) -> list:
self._last_audio_at = now
data = np.concatenate([self._pending, samples.astype(np.float32, copy=False)])
n = len(data) // WINDOW * WINDOW
self._pending = data[n:]
events: list = []
for i in range(0, n, WINDOW):
events += self._window(data[i:i + WINDOW], now)
if self.open_utt is not None and len(self._win) > self._last_partial_len \
and now - self._last_partial_at >= self.settings.partial_interval_ms / 1000:
self._last_partial_at = now
self._last_partial_len = len(self._win)
events.append(Partial(self._utt_count, np.concatenate(self._win)))
return events
def _window(self, w: np.ndarray, now: float) -> list:
p = self.vad.prob(w)
if not self._active:
if p >= START_PROB:
self._active = True
self._win = self._pre + [w]
self._pre = []
self._speech = 1
self._trailing = 0
else:
self._pre.append(w)
del self._pre[:-_windows(PRE_ROLL_MS)]
return []
self._win.append(w)
if p >= KEEP_PROB:
self._speech += 1
self._trailing = 0
else:
self._trailing += 1
events: list = []
if not self._started and self._speech * WINDOW_MS >= MIN_SPEECH_MS:
self._started = True
self._utt_count += 1
self._last_partial_at = now
events.append(SpeechStart(self._utt_count))
if self._trailing * WINDOW_MS >= self.settings.end_silence_ms:
events += self._end("silence", now)
elif len(self._win) * WINDOW_MS >= MAX_UTTERANCE_MS:
events += self._end("max_length", now)
return events
def _end(self, reason: str, now: float) -> list:
events: list = []
if self._active and self._started:
drop = max(0, self._trailing - _windows(TAIL_KEEP_MS))
kept = self._win[:len(self._win) - drop]
events.append(Final(self._utt_count, np.concatenate(kept), reason, now))
self._reset()
return events
def speaking_end(self, now: float) -> list:
self._pending = np.zeros(0, dtype=np.float32)
return self._end("speaking_end", now) if self._active else []
def tick(self, now: float) -> list:
if self._active and self._last_audio_at is not None \
and now - self._last_audio_at >= self.settings.end_silence_ms / 1000:
self._pending = np.zeros(0, dtype=np.float32)
return self._end("idle", now)
return []

View File

@@ -0,0 +1,76 @@
"""Punctuation-only sentence splitting for streaming TTS (language-agnostic).
Same rule as the v1 bridge (``bridge/text_utils.py``), which is legacy and goes away with v2; the
worker keeps its own copy so it does not depend on the v1 package.
"""
from __future__ import annotations
import re
# A boundary is a run of newlines, a run of CJK terminators (always a sentence end, CJK puts no space
# after one), or a run of ASCII terminators that really ends a sentence: followed by whitespace, a
# closing quote or bracket, or the end of the text. Dots inside tokens (17.5, v2.0, example.com) are
# followed by a non-space character, so they never match.
_BOUNDARY = re.compile(
r"""
(?P<nl>\n+)
| (?P<cjk>[。!?…]+)
| (?P<ascii>[.!?]+)(?=[)\]"'”’》」』]*(?:\s|$))
""",
re.VERBOSE,
)
def split_sentences(text: str | None, min_len: int = 5) -> list[str]:
"""Split ``text`` into sentence chunks; fragments under ``min_len`` characters join a neighbour."""
return _split(text, min_len, merge_short_tail=True)
def _split(text: str | None, min_len: int, merge_short_tail: bool) -> list[str]:
text = (text or "").strip()
if not text:
return []
chunks: list[str] = []
buf = ""
last = 0
for m in _BOUNDARY.finditer(text):
buf += text[last:m.end()]
last = m.end()
if len(buf.strip()) >= min_len:
chunks.append(buf.strip())
buf = ""
tail = (buf + text[last:]).strip()
if tail:
if merge_short_tail and chunks and len(tail) < min_len:
chunks[-1] = chunks[-1] + " " + tail
else:
chunks.append(tail)
return chunks
class StreamingSplitter:
"""Feed text as it streams in; get back every sentence that is certainly complete."""
def __init__(self, min_len: int = 5):
self.min_len = min_len
self._buf = ""
def feed(self, text: str) -> list[str]:
self._buf += text
# The tail may still grow, so it is not merged backwards here.
chunks = _split(self._buf, self.min_len, merge_short_tail=False)
if len(chunks) <= 1:
return []
done = chunks[:-1]
# Keep the raw text after the last emitted chunk, so spacing inside the tail is untouched.
consumed = 0
for c in done:
consumed = self._buf.index(c, consumed) + len(c)
self._buf = self._buf[consumed:]
return done
def finish(self) -> list[str]:
rest = self._buf.strip()
self._buf = ""
return [rest] if rest else []

310
src/voice_worker/server.py Normal file
View File

@@ -0,0 +1,310 @@
"""Local WebSocket server: one session per connection (protocol in voice_worker.spec.md)."""
from __future__ import annotations
import asyncio
import hmac
import ipaddress
import itertools
import json
import time
from dataclasses import dataclass
from typing import Callable
import numpy as np
from jarvis.debug import debug_log
from .audio import InputConverter
from .protocol import KIND_AUDIO, ProtocolError, decode_audio_frame, encode_tts_frame
from .scheduler import DecodeOptions, SttResult, SttScheduler, Transcriber
from .segmenter import SR, Final, Partial, Segmenter, SpeechStart, VadStream
from .settings import SttSettings
from .tts import TtsEngine, TtsRequest, TtsService
CLOSE_BAD_HELLO = 4400
CLOSE_BAD_TOKEN = 4401
MAX_SPEAKERS = 32
def is_loopback(host: str) -> bool:
if host == "localhost":
return True
try:
return ipaddress.ip_address(host).is_loopback
except ValueError:
return False
class VoiceWorker:
def __init__(self, token: str, transcriber: Transcriber | None, vad_factory: Callable[[], VadStream],
tts_engine: TtsEngine | None = None, hello_timeout_s: float = 5.0, tick_s: float = 0.05):
if not token:
raise ValueError("a connection token is required")
self.token = token
self.transcriber = transcriber
self.vad_factory = vad_factory
self.tts_engine = tts_engine
self.hello_timeout_s = hello_timeout_s
self.tick_s = tick_s
self.scheduler: SttScheduler | None = None
self.tts: TtsService | None = None
self._ids = itertools.count(1)
async def start(self) -> None:
if self.transcriber is not None:
self.scheduler = SttScheduler(self.transcriber)
self.scheduler.start()
if self.tts_engine is not None:
self.tts = TtsService(self.tts_engine)
self.tts.start()
async def stop(self) -> None:
if self.scheduler:
await self.scheduler.stop()
if self.tts:
await self.tts.stop()
async def serve(self, host: str, port: int):
"""Start listening; returns the websockets server. Loopback addresses only."""
if not is_loopback(host):
raise ValueError("the voice worker only listens on a loopback address")
from websockets.asyncio.server import serve
await self.start()
return await serve(self.handle, host, port, max_size=2 ** 20)
async def handle(self, ws) -> None:
try:
first = await asyncio.wait_for(ws.recv(), self.hello_timeout_s)
hello = json.loads(first) if isinstance(first, str) else None
except (asyncio.TimeoutError, ValueError):
hello = None
except Exception:
return
if not isinstance(hello, dict) or hello.get("type") != "hello":
debug_log("connection refused: no hello", "voice_worker")
await ws.close(CLOSE_BAD_HELLO, "hello expected")
return
token = hello.get("token")
if not isinstance(token, str) or not hmac.compare_digest(token.encode(), self.token.encode()):
debug_log("connection refused: bad token", "voice_worker")
await ws.close(CLOSE_BAD_TOKEN, "bad token")
return
try:
session = Session(self, ws, next(self._ids), hello)
except ValueError as e:
await ws.close(CLOSE_BAD_HELLO, str(e)[:100])
return
await session.run()
@dataclass
class _Speaker:
converter: InputConverter
segmenter: Segmenter
def _format(value, default: dict, rates: tuple[int, ...] | None) -> dict:
if value is None:
return dict(default)
if not isinstance(value, dict):
raise ValueError("format must be an object")
rate, channels = value.get("rate", default["rate"]), value.get("channels", default["channels"])
ok_rate = isinstance(rate, int) and (rate in rates if rates else 8000 <= rate <= 96000)
if not ok_rate or channels not in (1, 2) or isinstance(channels, bool):
raise ValueError("unsupported audio format")
return {"rate": rate, "channels": channels}
class Session:
def __init__(self, worker: VoiceWorker, ws, sid: int, hello: dict):
self.worker = worker
self.ws = ws
self.sid = sid
self.audio = _format(hello.get("audio"), {"rate": 48000, "channels": 2}, (16000, 48000))
self.tts_format = _format(hello.get("tts"), {"rate": 48000, "channels": 2}, None)
self.settings = SttSettings()
stt = hello.get("stt")
if stt is not None:
if not isinstance(stt, dict):
raise ValueError("stt must be an object")
self.settings.update(stt)
self.speakers: dict[str, _Speaker] = {}
self.tts_requests: dict[str, TtsRequest] = {}
self.out: asyncio.Queue = asyncio.Queue()
self.closed = False
# --- outbound -------------------------------------------------------------------------------
def send(self, msg) -> None:
if not self.closed:
self.out.put_nowait(msg if isinstance(msg, bytes) else json.dumps(msg, ensure_ascii=False))
def error(self, code: str, message: str, **extra) -> None:
self.send({"type": "error", "code": code, "message": message, **extra})
async def _writer(self) -> None:
while True:
await self.ws.send(await self.out.get())
# --- lifecycle ------------------------------------------------------------------------------
async def run(self) -> None:
w = self.worker
debug_log(f"session {self.sid} connected audio={self.audio} tts={self.tts_format}", "voice_worker")
self.send({"type": "hello_ok", "stt_ready": w.scheduler is not None, "tts_ready": w.tts is not None,
"audio": self.audio, "tts": self.tts_format, "stt": self.settings.as_dict()})
writer = asyncio.create_task(self._writer())
ticker = asyncio.create_task(self._ticker())
try:
async for frame in self.ws:
if isinstance(frame, bytes):
self._on_binary(frame)
else:
self._on_text(frame)
except Exception as e: # connection dropped
debug_log(f"session {self.sid} receive ended: {type(e).__name__}", "voice_worker")
finally:
self.closed = True
ticker.cancel()
writer.cancel()
if w.scheduler:
w.scheduler.drop_partials(lambda key: key[0] == self.sid)
if w.tts:
for req in list(self.tts_requests.values()):
w.tts.cancel(req)
debug_log(f"session {self.sid} closed", "voice_worker")
async def _ticker(self) -> None:
while True:
await asyncio.sleep(self.worker.tick_s)
now = time.monotonic()
for speaker, st in list(self.speakers.items()):
self._handle_events(speaker, st, st.segmenter.tick(now))
# --- STT ------------------------------------------------------------------------------------
def _on_binary(self, frame: bytes) -> None:
if not frame or frame[0] != KIND_AUDIO:
self.error("bad_frame", "unknown binary frame kind")
return
try:
speaker, pcm = decode_audio_frame(frame)
except ProtocolError as e:
self.error("bad_frame", str(e))
return
st = self.speakers.get(speaker)
if st is None:
if len(self.speakers) >= MAX_SPEAKERS:
self.error("bad_frame", f"more than {MAX_SPEAKERS} speakers")
return
st = _Speaker(InputConverter(self.audio["rate"], self.audio["channels"]),
Segmenter(self.worker.vad_factory(), self.settings))
self.speakers[speaker] = st
samples = st.converter.push(pcm)
self._handle_events(speaker, st, st.segmenter.push(samples, time.monotonic()))
def _opts(self, beam: int) -> DecodeOptions:
return DecodeOptions(beam=beam, language=self.settings.language, hotwords=self.settings.hotwords)
def _handle_events(self, speaker: str, st: _Speaker, events: list) -> None:
sched = self.worker.scheduler
for ev in events:
if isinstance(ev, SpeechStart):
self.send({"type": "speech_start", "speaker": speaker, "utt": ev.utt})
elif isinstance(ev, Partial):
if sched and self.settings.partials:
sched.submit_partial((self.sid, speaker), ev.audio, self._opts(1),
self._partial_cb(speaker, st, ev.utt))
elif isinstance(ev, Final):
debug_log(f"utterance end speaker={speaker} utt={ev.utt} reason={ev.reason} "
f"ms={len(ev.audio) * 1000 // SR}", "voice_worker")
if sched is None:
self._send_final(speaker, ev, SttResult("", 0, "stt unavailable"))
else:
sched.submit_final((self.sid, speaker), ev.audio, self._opts(self.settings.final_beam),
lambda res, ev=ev: self._send_final(speaker, ev, res))
def _partial_cb(self, speaker: str, st: _Speaker, utt: int):
def cb(res: SttResult) -> None:
# Drop it when the utterance already ended or captions were turned off meanwhile.
if st.segmenter.open_utt == utt and self.settings.partials and not res.error:
self.send({"type": "partial", "speaker": speaker, "utt": utt, "text": res.text})
return cb
def _send_final(self, speaker: str, ev: Final, res: SttResult) -> None:
self.send({"type": "final", "speaker": speaker, "utt": ev.utt, "text": res.text,
"end_reason": ev.reason, "audio_ms": len(ev.audio) * 1000 // SR,
"wait_ms": int((time.monotonic() - ev.decided_at) * 1000), "stt_ms": res.stt_ms})
# --- text messages --------------------------------------------------------------------------
def _on_text(self, raw: str) -> None:
try:
msg = json.loads(raw)
except ValueError:
self.error("bad_message", "not JSON")
return
if not isinstance(msg, dict):
self.error("bad_message", "not an object")
return
kind = msg.get("type")
handler = {"speaking_end": self._on_speaking_end, "config": self._on_config, "tts": self._on_tts,
"tts_cancel": self._on_tts_cancel}.get(kind)
if handler is None:
self.error("bad_message", f"unknown type: {kind!r}")
return
handler(msg)
def _on_speaking_end(self, msg: dict) -> None:
st = self.speakers.get(msg.get("speaker"))
if st is not None:
self._handle_events(msg["speaker"], st, st.segmenter.speaking_end(time.monotonic()))
def _on_config(self, msg: dict) -> None:
changes = {k: v for k, v in msg.items() if k != "type"}
try:
self.settings.update(changes)
except ValueError as e:
self.error("bad_message", str(e))
return
if not self.settings.partials and self.worker.scheduler:
self.worker.scheduler.drop_partials(lambda key: key[0] == self.sid)
debug_log(f"session {self.sid} settings changed: {sorted(changes)}", "voice_worker")
def _on_tts(self, msg: dict) -> None:
rid, text, more = msg.get("id"), msg.get("text", ""), msg.get("more", False)
if not isinstance(rid, str) or not 0 < len(rid.encode()) < 256 or not isinstance(text, str) \
or not isinstance(more, bool):
self.error("bad_message", "tts needs a string id and text, and a boolean more")
return
tts = self.worker.tts
if tts is None:
self.error("tts_unavailable", "no TTS engine is loaded", id=rid)
return
req = self.tts_requests.get(rid)
if req is None:
req = TtsRequest(rid, self.tts_format["rate"], self._tts_chunk_cb(rid), self._tts_done,
self._tts_error)
self.tts_requests[rid] = req
elif req.closed:
self.error("bad_message", "this tts id is already closed", id=rid)
return
tts.feed(req, text, more)
def _on_tts_cancel(self, msg: dict) -> None:
req = self.tts_requests.get(msg.get("id"))
if req is not None and self.worker.tts:
self.worker.tts.cancel(req)
def _tts_chunk_cb(self, rid: str):
def cb(seq: int, pcm: np.ndarray) -> None:
pcm = np.asarray(pcm, dtype="<i2")
if self.tts_format["channels"] == 2:
pcm = np.repeat(pcm, 2)
self.send(encode_tts_frame(rid, seq, pcm.tobytes()))
return cb
def _tts_done(self, req: TtsRequest) -> None:
self.tts_requests.pop(req.id, None)
self.send({"type": "tts_done", "id": req.id, "chunks": req.chunks, "cancelled": req.cancelled,
"failed": req.failed})
def _tts_error(self, req: TtsRequest, message: str) -> None:
self.error("tts_failed", message, id=req.id)

View File

@@ -0,0 +1,60 @@
"""Per-connection STT settings, validated as a whole before any of them change."""
from __future__ import annotations
from dataclasses import asdict, dataclass, fields
@dataclass
class SttSettings:
partials: bool = True
partial_interval_ms: int = 400
end_silence_ms: int = 300
final_beam: int = 2
language: str | None = "ko"
hotwords: tuple[str, ...] = ()
def update(self, changes: dict) -> None:
"""Apply ``changes``; raises ValueError and changes nothing if any value is bad."""
names = {f.name for f in fields(self)}
staged = asdict(self)
for key, value in changes.items():
if key not in names:
raise ValueError(f"unknown STT setting: {key}")
staged[key] = _check(key, value)
for key, value in staged.items():
setattr(self, key, value)
def as_dict(self) -> dict:
d = asdict(self)
d["hotwords"] = list(self.hotwords)
return d
def _int_in(key: str, value, lo: int, hi: int) -> int:
if isinstance(value, bool) or not isinstance(value, int) or not lo <= value <= hi:
raise ValueError(f"{key} must be an integer in {lo}-{hi}")
return value
def _check(key: str, value):
if key == "partials":
if not isinstance(value, bool):
raise ValueError("partials must be true or false")
return value
if key == "partial_interval_ms":
return _int_in(key, value, 100, 5000)
if key == "end_silence_ms":
return _int_in(key, value, 100, 2000)
if key == "final_beam":
return _int_in(key, value, 1, 5)
if key == "language":
if value is not None and (not isinstance(value, str) or not 2 <= len(value) <= 8):
raise ValueError("language must be a language code or null")
return value
if key == "hotwords":
if (not isinstance(value, (list, tuple)) or len(value) > 50
or not all(isinstance(w, str) and 0 < len(w) <= 64 for w in value)):
raise ValueError("hotwords must be a list of up to 50 short strings")
return tuple(value)
raise ValueError(f"unknown STT setting: {key}") # pragma: no cover

View File

@@ -0,0 +1,78 @@
"""faster-whisper batch transcription: one encoder and one decoder pass for every clip in a batch."""
from __future__ import annotations
import glob
import os
import sys
import numpy as np
from jarvis.debug import debug_log
from .audio import peak_normalise
from .scheduler import DecodeOptions
SR = 16000
NO_SPEECH_THRESHOLD = 0.5 # same gate as v1 (whisper_no_speech_threshold)
MIN_CONFIDENCE = 0.3 # avg_logprob + 1, same as v1 (whisper_min_confidence)
def add_cuda_dll_dirs() -> None:
"""On Windows ctranslate2 loads cuBLAS/cuDNN from the PATH; the pip wheels put them in nvidia/*/bin."""
if not hasattr(os, "add_dll_directory"):
return
for sp in sys.path:
for d in glob.glob(os.path.join(sp, "nvidia", "*", "bin")):
if os.path.isdir(d):
os.environ["PATH"] = d + os.pathsep + os.environ.get("PATH", "")
os.add_dll_directory(d)
class WhisperTranscriber:
def __init__(self, model: str = "large-v3-turbo", device: str = "cuda", compute_type: str = "int8_float16"):
add_cuda_dll_dirs()
from faster_whisper import WhisperModel
self.model = WhisperModel(model, device=device, compute_type=compute_type)
self._tokenizers: dict = {}
debug_log(f"whisper loaded model={model} device={device} compute={compute_type}", "voice_worker")
def _tokenizer(self, language: str):
tok = self._tokenizers.get(language)
if tok is None:
from faster_whisper.tokenizer import Tokenizer
tok = Tokenizer(self.model.hf_tokenizer, self.model.model.is_multilingual, task="transcribe",
language=language)
self._tokenizers[language] = tok
return tok
def warm_up(self) -> None:
self.transcribe([np.zeros(SR, dtype=np.float32)], DecodeOptions(beam=1, language="en", hotwords=()))
def transcribe(self, clips: list[np.ndarray], opts: DecodeOptions) -> list[str]:
m = self.model
n_frames = m.feature_extractor.nb_max_frames
feats = []
for c in clips:
a = peak_normalise(c[: SR * 30])
a = np.pad(a, (0, SR * 30 - len(a)))
feats.append(m.feature_extractor(a, padding=0)[:, :n_frames])
enc = m.encode(np.stack(feats).astype(np.float32))
if opts.language:
langs = [opts.language] * len(clips)
else:
langs = [r[0][0][2:-2] for r in m.model.detect_language(enc)]
hot = " ".join(opts.hotwords) or None
prompts = [m.get_prompt(self._tokenizer(lang), [], without_timestamps=True, hotwords=hot) for lang in langs]
results = m.model.generate(enc, prompts, beam_size=opts.beam, max_length=224,
suppress_blank=True, suppress_tokens=[-1], return_scores=True,
return_no_speech_prob=True)
texts = []
for lang, r in zip(langs, results):
tokens = r.sequences_ids[0]
avg_logprob = r.scores[0] * len(tokens) / (len(tokens) + 1)
if r.no_speech_prob >= NO_SPEECH_THRESHOLD or avg_logprob + 1 < MIN_CONFIDENCE:
texts.append("")
else:
texts.append(self._tokenizer(lang).decode(tokens).strip())
return texts

105
src/voice_worker/tts.py Normal file
View File

@@ -0,0 +1,105 @@
"""Sentence-by-sentence TTS shared by every connection, first come first served."""
from __future__ import annotations
import asyncio
import time
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
from typing import Callable, Protocol
import numpy as np
from jarvis.debug import debug_log
from .sentences import StreamingSplitter
class TtsEngine(Protocol):
def synthesize(self, text: str, rate: int) -> np.ndarray:
"""Return int16 mono samples at ``rate``."""
...
@dataclass
class TtsRequest:
id: str
rate: int
on_chunk: Callable[[int, np.ndarray], None]
on_done: Callable[["TtsRequest"], None]
on_error: Callable[["TtsRequest", str], None]
splitter: StreamingSplitter = field(default_factory=StreamingSplitter)
next_seq: int = 0
outstanding: int = 0
chunks: int = 0
closed: bool = False
cancelled: bool = False
failed: bool = False
finished: bool = False
class TtsService:
def __init__(self, engine: TtsEngine):
self.engine = engine
self._queue: asyncio.Queue = asyncio.Queue()
self._executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="tts")
self._task: asyncio.Task | None = None
def start(self) -> None:
self._task = asyncio.get_running_loop().create_task(self._run())
async def stop(self) -> None:
if self._task:
self._task.cancel()
try:
await self._task
except asyncio.CancelledError:
pass
self._executor.shutdown(wait=False, cancel_futures=True)
def feed(self, req: TtsRequest, text: str, more: bool) -> None:
sentences = req.splitter.feed(text)
if not more:
sentences += req.splitter.finish()
req.closed = True
for s in sentences:
req.outstanding += 1
self._queue.put_nowait((req, req.next_seq, s))
req.next_seq += 1
self._maybe_done(req)
def cancel(self, req: TtsRequest) -> None:
req.cancelled = True
req.closed = True
debug_log(f"tts cancel id={req.id} sent={req.chunks}", "voice_worker")
self._maybe_done(req)
def _maybe_done(self, req: TtsRequest) -> None:
if req.closed and req.outstanding == 0 and not req.finished:
req.finished = True
req.on_done(req)
async def _run(self) -> None:
loop = asyncio.get_running_loop()
while True:
req, seq, text = await self._queue.get()
try:
if req.cancelled or req.failed:
continue
t0 = time.perf_counter()
try:
pcm = await loop.run_in_executor(self._executor, self.engine.synthesize, text, req.rate)
except Exception as e:
req.failed = True
req.closed = True
debug_log(f"tts failed id={req.id} seq={seq}: {type(e).__name__}: {e}", "voice_worker")
req.on_error(req, f"{type(e).__name__}: {e}")
continue
debug_log(f"tts sentence id={req.id} seq={seq} chars={len(text)} "
f"ms={int((time.perf_counter() - t0) * 1000)}", "voice_worker")
if not req.cancelled:
req.chunks += 1
req.on_chunk(seq, pcm)
finally:
req.outstanding -= 1
self._maybe_done(req)

View File

@@ -0,0 +1,65 @@
"""TTS engine that drives ``melo_child.py`` in the MeloTTS venv over stdin/stdout."""
from __future__ import annotations
import json
import os
import struct
import subprocess
import sys
import threading
import numpy as np
from jarvis.debug import debug_log
CHILD = os.path.join(os.path.dirname(os.path.abspath(__file__)), "melo_child.py")
class MeloEngine:
def __init__(self, python: str, device: str = "cuda", speed: float = 1.5, language: str = "KR"):
self.cmd = [python, CHILD, "--device", device, "--speed", str(speed), "--language", language]
self._lock = threading.Lock()
self._proc: subprocess.Popen | None = None
self.native_rate: int | None = None
self._start()
def _start(self) -> None:
self._proc = subprocess.Popen(self.cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=None, creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
header, _ = self._read()
if not header.get("ready"):
self.close()
raise RuntimeError(f"MeloTTS failed to start: {header.get('error')}")
self.native_rate = header["rate"]
debug_log(f"melo ready native_rate={self.native_rate}", "voice_worker")
def _read(self) -> tuple[dict, bytes]:
out = self._proc.stdout
raw = out.read(4)
if len(raw) < 4:
raise RuntimeError("MeloTTS child exited")
(n,) = struct.unpack("<I", raw)
header = json.loads(out.read(n))
payload = out.read(header.get("samples", 0) * 2) if header.get("ok") else b""
return header, payload
def synthesize(self, text: str, rate: int) -> np.ndarray:
with self._lock:
if self._proc is None or self._proc.poll() is not None:
debug_log("melo child not running, restarting", "voice_worker")
self._start()
self._proc.stdin.write((json.dumps({"text": text, "rate": rate}) + "\n").encode())
self._proc.stdin.flush()
header, payload = self._read()
if not header.get("ok"):
raise RuntimeError(header.get("error", "MeloTTS failed"))
return np.frombuffer(payload, dtype="<i2")
def close(self) -> None:
if self._proc and self._proc.poll() is None:
self._proc.stdin.close()
try:
self._proc.wait(5)
except subprocess.TimeoutExpired:
self._proc.kill()

45
src/voice_worker/vad.py Normal file
View File

@@ -0,0 +1,45 @@
"""Streaming Silero VAD (the v6 model bundled with faster-whisper), one state per speaker."""
from __future__ import annotations
import os
import numpy as np
from .segmenter import WINDOW
_CONTEXT = 64
class SileroVad:
"""Owns the ONNX session; ``stream()`` gives each speaker its own recurrent state."""
def __init__(self, path: str | None = None):
import onnxruntime
if path is None:
from faster_whisper.utils import get_assets_path
path = os.path.join(get_assets_path(), "silero_vad_v6.onnx")
opts = onnxruntime.SessionOptions()
opts.inter_op_num_threads = 1
opts.intra_op_num_threads = 1
opts.enable_cpu_mem_arena = False
opts.log_severity_level = 4
self.session = onnxruntime.InferenceSession(path, providers=["CPUExecutionProvider"], sess_options=opts)
def stream(self) -> "SileroStream":
return SileroStream(self.session)
class SileroStream:
def __init__(self, session):
self.session = session
self.h = np.zeros((1, 1, 128), dtype=np.float32)
self.c = np.zeros((1, 1, 128), dtype=np.float32)
self.context = np.zeros(_CONTEXT, dtype=np.float32)
def prob(self, window: np.ndarray) -> float:
assert len(window) == WINDOW
x = np.concatenate([self.context, window.astype(np.float32, copy=False)])[None, :]
out, self.h, self.c = self.session.run(None, {"input": x, "h": self.h, "c": self.c})
self.context = x[0, -_CONTEXT:]
return float(np.asarray(out).reshape(-1)[0])

View File

@@ -0,0 +1,148 @@
# Voice worker spec (v2 M1)
The voice worker is the Python process that turns many Discord speakers' audio into text and the
assistant's reply text into speech. The Electron app's Discord voice core talks to it over one local
WebSocket. Design background lives in `docs/v2/voice-and-llm.md` and `docs/v2/architecture.md`.
## Principles
- **Local and private.** The server binds to a loopback address only and refuses any other host. Every
connection must present the token the app generated; the token comes from the environment
(`JARVIS_VOICE_TOKEN`), never from the command line, so it does not show in process listings. Logs
carry timings, sizes and counts, never transcript or reply text.
- **One shared model.** A single faster-whisper model serves every speaker. Speakers that need
transcription at the same moment go through the model in one batch.
- **Finals before partials.** A finished utterance is always transcribed before any live caption.
- **Language-agnostic.** Nothing in the worker matches words. The STT language is a setting (`null`
means detect per utterance); sentence splitting uses punctuation only.
- **Engines are pluggable.** STT, VAD and TTS sit behind small interfaces so the logic is tested with
fakes and the real models are tested separately.
## Running
```
JARVIS_VOICE_TOKEN=<secret> python -m voice_worker --host 127.0.0.1 --port 8771 \
[--stt-model large-v3-turbo] [--stt-device cuda] [--stt-compute int8_float16] \
[--melo-python <path to the MeloTTS venv python>] [--melo-device cuda] [--melo-speed 1.5]
```
Run from `src/` (or with `src/` on `PYTHONPATH`) in a venv built from `voice_worker/requirements.txt`. MeloTTS needs its own Python 3.11 venv
(transformers 4.27), so it runs as a child process (`voice_worker/melo_child.py`) under
`--melo-python`. Without `--melo-python` the worker has no TTS and answers `tts` requests with an
error. Both models load and warm up once at start; `hello_ok` reports whether each is ready.
## Protocol
Text frames are JSON objects with a `type`. Binary frames start with a one-byte kind.
### Handshake
The first frame must be `hello` within 5 seconds, otherwise the worker closes with code 4400. A wrong
token closes with 4401.
```
-> {"type": "hello", "token": "...",
"audio": {"rate": 48000, "channels": 2}, // incoming PCM format: rate 16000|48000, channels 1|2
"tts": {"rate": 48000, "channels": 2}, // outgoing PCM format (any rate, channels 1|2)
"stt": { ...settings, see config... }} // optional
<- {"type": "hello_ok", "stt_ready": true, "tts_ready": true,
"audio": {...}, "tts": {...}, "stt": { ...effective settings... }}
```
### STT
```
-> binary 0x01 | u8 n | speaker id (n bytes UTF-8) | PCM16LE samples in the hello "audio" format
-> {"type": "speaking_end", "speaker": "..."} // Discord speaking stopped for this speaker
-> {"type": "config", ...any STT settings...} // takes effect immediately
<- {"type": "speech_start", "speaker": "...", "utt": 3}
<- {"type": "partial", "speaker": "...", "utt": 3, "text": "..."}
<- {"type": "final", "speaker": "...", "utt": 3, "text": "...", "end_reason": "silence",
"audio_ms": 2310, "wait_ms": 160, "stt_ms": 120}
```
`utt` counts utterances per speaker. `final` always closes an utterance that sent `speech_start`, even
when the text is empty (noise that Whisper rejected). `wait_ms` is from the moment the end was decided
to the moment the final was sent; `stt_ms` is the model time of the batch it went through.
STT settings (all optional in `hello` and `config`):
| Setting | Default | Meaning |
|---|---|---|
| `partials` | `true` | Send live captions. `false` drops queued partials and stops new ones; finals continue. The app turns it off while the LLM answers during a game. |
| `partial_interval_ms` | 400 | At most one partial per speaker per interval, only when audio grew. |
| `end_silence_ms` | 300 | Trailing VAD silence that ends an utterance. Range 100-2000. |
| `final_beam` | 2 | Beam size for finals (partials always use 1). Range 1-5. |
| `language` | `"ko"` | Whisper language code, or `null` to detect per utterance. |
| `hotwords` | `[]` | Words (wake words, names) passed to Whisper as a hint. |
### Utterance rules
Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filter, and run through
its own streaming Silero VAD (512-sample windows, state kept between frames).
- **Start**: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
- **speech_start** is sent once the utterance holds 250 ms of speech windows. An utterance that ends
before that is dropped without any message (a click or a cough).
- **Speech continues** while windows stay at or above 0.35 (hysteresis); a quieter window counts as
silence.
- **End**, with `end_reason`:
- `silence`: trailing silence reached `end_silence_ms`.
- `speaking_end`: the app sent `speaking_end`. Discord stops sending packets when a user stops
talking, so the worker cannot wait for silence that never arrives.
- `idle`: no audio arrived for `end_silence_ms` (a missed `speaking_end`).
- `max_length`: the utterance reached 28 seconds (Whisper's window). The next audio starts a new
utterance.
- The final clip keeps at most 200 ms of trailing silence.
- Before transcription the clip is peak-normalised to 0.9 with the gain capped at 10x.
### Scheduling
One thread owns the STT model and loops:
1. If any finals are waiting, take all of them (up to 16) and transcribe in one batch with
`final_beam`.
2. Otherwise take up to 2 waiting partials and transcribe them in one batch with beam 1. A speaker
has at most one waiting partial; a newer request replaces the older one and goes to the back of the
line, so speakers take turns.
Whisper's encoder always runs on a full 30-second window, so its cost grows with every clip in a batch
(measured on an RTX 5050: 0.2 s for one clip, 2.1 s for ten). A large partial batch would make a final
wait seconds behind it; two clips keep that wait to about one clip's time. With many people talking,
captions fall behind and only the newest audio per speaker is transcribed.
Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has
already ended, or that is older than the last partial sent, is dropped. Whisper output is rejected
(empty text) when `no_speech_prob >= 0.5` or the confidence (`avg_logprob + 1`) is below 0.3, the
same gate as v1.
### TTS
```
-> {"type": "tts", "id": "r1", "text": "...", "more": false} // more=true: more text follows for r1
-> {"type": "tts_cancel", "id": "r1"}
<- binary 0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}
```
Text is split into sentences on punctuation only: a run of newlines, CJK terminators (`。!?…`), or
ASCII terminators followed by whitespace, a closing quote or bracket, or the end of the text, so
decimals, versions and URLs stay whole. Fragments under 5 characters are merged into a neighbour.
With `more=true` the worker synthesises every complete sentence at once and holds the unfinished tail
until more text or `more=false` arrives (a short last fragment is then spoken on its own, because the
sentence before it has already gone out). One audio chunk is sent per sentence, in order, as soon as it
is ready. Requests are served first come, first served by one TTS engine. `tts_cancel` drops the
request's unsent sentences, and `tts_done` follows with `cancelled: true`. A sentence that fails to
synthesise is reported with `error` (`code: "tts_failed"`) and the request ends with `failed: true`.
### Errors
`{"type": "error", "code": "...", "message": "...", "id": "..."}`. Codes: `bad_message` (unknown type,
bad JSON, bad field), `bad_frame` (malformed binary frame), `tts_unavailable`, `tts_failed`. Errors
never close the connection after the handshake.
## Debug logging
`debug_log(..., "voice_worker")` at: connection accepted or refused, settings changed, utterance end
(speaker, reason, length), each STT batch (kind, size, model time), each TTS sentence (length, time),
cancel, and engine failures.

View File

@@ -0,0 +1,122 @@
"""Voice worker: binary frames, input conversion to 16 kHz mono, clip normalisation."""
from __future__ import annotations
import numpy as np
import pytest
from voice_worker.audio import InputConverter, peak_normalise
from voice_worker.protocol import (KIND_AUDIO, KIND_TTS, ProtocolError, decode_audio_frame, encode_audio_frame,
encode_tts_frame, decode_tts_frame)
def _pcm16(x: np.ndarray) -> bytes:
return (np.clip(x, -1, 1) * 32767).astype("<i2").tobytes()
def _dominant_hz(x: np.ndarray, rate: int) -> float:
spec = np.abs(np.fft.rfft(x * np.hanning(len(x))))
return float(np.fft.rfftfreq(len(x), 1 / rate)[int(np.argmax(spec))])
@pytest.mark.unit
def test_audio_frame_round_trip_keeps_speaker_and_pcm():
pcm = _pcm16(np.linspace(-0.5, 0.5, 960))
frame = encode_audio_frame("123456789012345678", pcm)
assert frame[0] == KIND_AUDIO
speaker, payload = decode_audio_frame(frame)
assert speaker == "123456789012345678"
assert payload == pcm
@pytest.mark.unit
def test_tts_frame_round_trip_keeps_id_and_sequence():
pcm = b"\x01\x00\x02\x00"
frame = encode_tts_frame("reply-7", 3, pcm)
assert frame[0] == KIND_TTS
assert decode_tts_frame(frame) == ("reply-7", 3, pcm)
@pytest.mark.unit
@pytest.mark.parametrize("frame", [b"", b"\x01", b"\x09\x01a\x00\x00", b"\x01\x05ab", b"\x01\x01a\x00",
b"\x01\x02\xff\xfe\x00\x00"])
def test_malformed_audio_frames_are_rejected(frame):
with pytest.raises(ProtocolError):
decode_audio_frame(frame)
@pytest.mark.unit
def test_48k_stereo_becomes_16k_mono_and_keeps_the_tone():
rate = 48000
t = np.arange(rate) / rate
tone = 0.5 * np.sin(2 * np.pi * 1000 * t)
stereo = np.stack([tone, tone], axis=1).reshape(-1)
out = InputConverter(rate, 2).push(_pcm16(stereo))
assert out.dtype == np.float32
assert abs(len(out) - rate // 3) <= 1
assert abs(_dominant_hz(out, 16000) - 1000) < 5
assert 0.4 < np.max(np.abs(out[500:])) < 0.6
@pytest.mark.unit
def test_tones_above_the_new_nyquist_are_filtered_out():
rate = 48000
t = np.arange(rate) / rate
high = 0.5 * np.sin(2 * np.pi * 11000 * t)
out = InputConverter(rate, 1).push(_pcm16(high))
# At least 30 dB down instead of folding back to 5 kHz.
assert np.sqrt(np.mean(out[500:] ** 2)) < 0.5 / np.sqrt(2) * 10 ** (-30 / 20)
@pytest.mark.unit
def test_streaming_in_20ms_frames_matches_one_shot_conversion():
rng = np.random.default_rng(1)
sig = rng.uniform(-0.5, 0.5, 48000 * 2 // 2 * 2)
one = InputConverter(48000, 2).push(_pcm16(sig))
conv = InputConverter(48000, 2)
raw = _pcm16(sig)
step = 960 * 2 * 2 # 20 ms of 48 kHz stereo PCM16
parts = [conv.push(raw[i:i + step]) for i in range(0, len(raw), step)]
assert np.allclose(np.concatenate(parts), one, atol=1e-6)
@pytest.mark.unit
def test_16k_mono_passes_through():
sig = np.linspace(-0.25, 0.25, 1600)
out = InputConverter(16000, 1).push(_pcm16(sig))
assert np.allclose(out, sig, atol=1e-4)
@pytest.mark.unit
def test_unsupported_input_formats_are_refused():
with pytest.raises(ValueError):
InputConverter(44100, 2)
with pytest.raises(ValueError):
InputConverter(48000, 6)
@pytest.mark.unit
def test_odd_byte_counts_are_carried_to_the_next_frame():
sig = _pcm16(np.full(10, 0.5))
conv = InputConverter(16000, 1)
a = conv.push(sig[:5])
b = conv.push(sig[5:])
assert len(a) + len(b) == 10
@pytest.mark.unit
def test_quiet_clip_is_raised_to_the_target_peak():
x = np.full(100, 0.2, dtype=np.float32)
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)
@pytest.mark.unit
def test_near_silence_is_not_boosted_past_the_gain_cap():
x = np.full(100, 0.001, dtype=np.float32)
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.01)
@pytest.mark.unit
def test_loud_clip_is_brought_down_to_the_target_peak():
x = np.array([0.0, 1.0, -0.5], dtype=np.float32)
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)

View File

@@ -0,0 +1,209 @@
"""Voice worker with the real models: ten recorded clips streamed at once, and a TTS round trip.
Opt-in because it needs a CUDA GPU and the downloaded models:
JARVIS_VOICE_REAL_TESTS=1 [JARVIS_MELO_PYTHON=<MeloTTS venv python>] pytest tests/test_voice_worker_real.py -s
On Linux, put the pip CUDA libraries on LD_LIBRARY_PATH (nvidia/cublas/lib, nvidia/cudnn/lib).
"""
from __future__ import annotations
import asyncio
import json
import os
import re
import time
import wave
from pathlib import Path
import numpy as np
import pytest
pytestmark = [pytest.mark.integration,
pytest.mark.skipif(os.environ.get("JARVIS_VOICE_REAL_TESTS") != "1",
reason="set JARVIS_VOICE_REAL_TESTS=1 to run on the GPU")]
SAMPLES = Path(__file__).resolve().parent.parent / "scripts" / "m0" / "assets" / "ko_samples"
TOKEN = "real-test-token"
FRAME_MS = 20
def cer(ref: str, hyp: str) -> float:
a, b = (re.sub(r"[\W_]+", "", s) for s in (ref, hyp))
prev = list(range(len(b) + 1))
for i, ca in enumerate(a, 1):
cur = [i]
for j, cb in enumerate(b, 1):
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
prev = cur
return prev[-1] / max(1, len(a))
def load(name: str) -> np.ndarray:
with wave.open(str(SAMPLES / name), "rb") as w:
assert w.getframerate() == 16000 and w.getnchannels() == 1
return np.frombuffer(w.readframes(w.getnframes()), dtype="<i2").astype(np.float32) / 32768.0
def to_48k_stereo(x16: np.ndarray) -> np.ndarray:
t48 = np.arange(len(x16) * 3) / 3
x48 = np.interp(t48, np.arange(len(x16)), x16)
return np.repeat(x48, 2)
@pytest.fixture(scope="module")
def engines():
from voice_worker.stt_whisper import WhisperTranscriber
from voice_worker.vad import SileroVad
stt = WhisperTranscriber()
stt.warm_up()
tts = None
if os.environ.get("JARVIS_MELO_PYTHON"):
from voice_worker.tts_melo import MeloEngine
tts = MeloEngine(os.environ["JARVIS_MELO_PYTHON"])
yield stt, SileroVad(), tts
if tts:
tts.close()
async def with_worker(engines, body):
from websockets.asyncio.client import connect
from voice_worker.server import VoiceWorker
stt, vad, tts = engines
worker = VoiceWorker(TOKEN, stt, vad.stream, tts)
server = await worker.serve("127.0.0.1", 0)
url = f"ws://127.0.0.1:{server.sockets[0].getsockname()[1]}"
try:
async with connect(url, max_size=2 ** 24) as ws:
await ws.send(json.dumps({"type": "hello", "token": TOKEN, "stt": {"hotwords": ["자비스"]}}))
assert json.loads(await ws.recv())["type"] == "hello_ok"
return await asyncio.wait_for(body(ws), 120)
finally:
server.close()
await server.wait_closed()
await worker.stop()
def test_ten_speakers_at_once(engines):
"""Ten speakers talk over each other in real time (48 kHz stereo, 20 ms frames, staggered starts).
Half end with Discord's speaking_end, half keep sending silence so only the VAD can end them."""
from voice_worker.protocol import encode_audio_frame
refs = json.loads((SAMPLES / "refs.json").read_text(encoding="utf-8"))["samples"]
files = [refs[i % len(refs)] for i in range(10)]
speakers = [f"spk{i}" for i in range(10)]
frame = 48000 * FRAME_MS // 1000 * 2 # stereo samples per frame
def track(i: int) -> list[bytes]:
lead = np.zeros(int(0.15 * i * 16000), dtype=np.float32)
tail = np.zeros(int(0.6 * 16000), dtype=np.float32) if i % 2 else np.zeros(0, dtype=np.float32)
clip = load(files[i]["file"])
if i % 2 == 0: # these end with speaking_end: Discord stops sending right after the last word
loud = np.nonzero(np.abs(clip) > 0.02)[0]
clip = clip[: loud[-1] + 800]
x = to_48k_stereo(np.concatenate([lead, clip, tail]))
pcm = (np.clip(x, -1, 1) * 32767).astype("<i2")
return [pcm[k:k + frame].tobytes() for k in range(0, len(pcm), frame)]
tracks = [track(i) for i in range(10)]
async def body(ws):
finals: dict[str, list[dict]] = {s: [] for s in speakers}
partials = 0
finished = asyncio.Event()
async def reader():
nonlocal partials
async for raw in ws:
msg = json.loads(raw)
if msg["type"] == "speech_start":
open_utts[msg["speaker"]] = msg["utt"]
elif msg["type"] == "partial":
partials += 1
elif msg["type"] == "final":
finals[msg["speaker"]].append(msg)
if sent_all.is_set() and all(finals.values()) and closed_all():
finished.set()
open_utts: dict[str, int] = {}
def closed_all() -> bool:
return all(len(finals[s]) >= open_utts.get(s, 0) for s in speakers)
sent_all = asyncio.Event()
rd = asyncio.create_task(reader())
t0 = time.monotonic()
for k in range(max(len(t) for t in tracks)):
for i, s in enumerate(speakers):
if k < len(tracks[i]):
await ws.send(encode_audio_frame(s, tracks[i][k]))
elif k == len(tracks[i]) and i % 2 == 0:
await ws.send(json.dumps({"type": "speaking_end", "speaker": s}))
await asyncio.sleep(max(0.0, t0 + (k + 1) * FRAME_MS / 1000 - time.monotonic()))
sent_all.set()
try:
await asyncio.wait_for(finished.wait(), 20)
except asyncio.TimeoutError:
pass
rd.cancel()
return finals, partials
finals, partials = asyncio.run(with_worker(engines, body))
rows = []
for i, s in enumerate(speakers):
text = " ".join(m["text"] for m in finals[s]).strip()
rows.append((s, files[i]["file"], len(finals[s]), cer(files[i]["text"], text),
[m["end_reason"] for m in finals[s]], [m["wait_ms"] for m in finals[s]], text))
for r in rows:
print(*r)
waits = [w for r in rows for w in r[5]]
print(f"partials={partials} wait_ms median={np.median(waits):.0f} max={max(waits)}")
assert all(r[2] >= 1 for r in rows), "every speaker gets a final"
assert np.mean([r[3] for r in rows]) <= 0.15
assert max(r[3] for r in rows) <= 0.35
assert {"speaking_end", "silence"} <= {reason for r in rows for reason in r[4]}
assert partials > 0
def test_tts_reply_comes_back_as_the_same_words(engines):
"""Speech the worker synthesises, fed back in as a speaker, is transcribed to the same text."""
if engines[2] is None:
pytest.skip("set JARVIS_MELO_PYTHON to the MeloTTS venv python")
from voice_worker.protocol import decode_tts_frame, encode_audio_frame
text = "오늘 서울은 맑고 기온은 이십삼 도예요. 우산은 필요 없어요."
async def body(ws):
await ws.send(json.dumps({"type": "config", "partials": False}))
await ws.send(json.dumps({"type": "tts", "id": "t1", "text": text}))
chunks = []
t0 = time.monotonic()
first = None
while True:
raw = await ws.recv()
if isinstance(raw, bytes):
chunks.append(decode_tts_frame(raw))
first = first or time.monotonic() - t0
elif json.loads(raw)["type"] == "tts_done":
break
pcm = b"".join(c[2] for c in chunks)
step = 48000 * FRAME_MS // 1000 * 2 * 2
for k in range(0, len(pcm), step):
await ws.send(encode_audio_frame("echo", pcm[k:k + step]))
await ws.send(json.dumps({"type": "speaking_end", "speaker": "echo"}))
got = []
# The pause between the sentences may split them, and the clip's own trailing silence may end
# the last one before speaking_end does, so collect finals until the worker goes quiet.
while True:
try:
msg = json.loads(await asyncio.wait_for(ws.recv(), 3))
except asyncio.TimeoutError:
break
if msg["type"] == "final":
got.append(msg["text"])
return chunks, first, " ".join(got)
chunks, first, heard = asyncio.run(with_worker(engines, body))
print(f"tts chunks={len(chunks)} first_chunk_s={first:.2f} heard={heard}")
assert [c[1] for c in chunks] == [0, 1]
assert cer(text, heard) <= 0.2

View File

@@ -0,0 +1,253 @@
"""Voice worker: the shared-model STT scheduler (batching, finals first, partial coalescing)."""
from __future__ import annotations
import asyncio
import threading
import numpy as np
import pytest
from voice_worker.scheduler import DecodeOptions, SttScheduler
FINAL = DecodeOptions(beam=2, language="ko", hotwords=())
PARTIAL = DecodeOptions(beam=1, language="ko", hotwords=())
class FakeTranscriber:
"""Names each clip by its length; the first call can be held to build up a queue."""
def __init__(self, hold_first: bool = False, fail_on_beam: int | None = None):
self.calls: list[tuple[DecodeOptions, list[int]]] = []
self.gate = threading.Event()
self.entered = threading.Event()
if not hold_first:
self.gate.set()
self.fail_on_beam = fail_on_beam
def transcribe(self, clips, opts):
self.entered.set()
self.gate.wait(5)
self.calls.append((opts, [len(c) for c in clips]))
if opts.beam == self.fail_on_beam:
raise RuntimeError("boom")
return [f"n{len(c)}" for c in clips]
def clip(n: int) -> np.ndarray:
return np.zeros(n, dtype=np.float32)
async def _run(body):
return await asyncio.wait_for(body(), 10)
def run(body):
return asyncio.run(_run(body))
async def _wait_entered(t: FakeTranscriber):
while not t.entered.is_set():
await asyncio.sleep(0.005)
@pytest.mark.unit
def test_finals_queued_while_busy_go_in_one_batch_before_partials():
t = FakeTranscriber(hold_first=True)
async def body():
s = SttScheduler(t, max_partial_batch=4)
s.start()
got = []
done = asyncio.Event()
def cb(name):
def _cb(res):
got.append((name, res.text))
if len(got) == 5:
done.set()
return _cb
s.submit_partial(("a", "1"), clip(10), PARTIAL, cb("p-first"))
await _wait_entered(t)
s.submit_partial(("a", "2"), clip(20), PARTIAL, cb("p2"))
s.submit_final(("a", "3"), clip(30), FINAL, cb("f3"))
s.submit_final(("a", "4"), clip(40), FINAL, cb("f4"))
s.submit_partial(("a", "5"), clip(50), PARTIAL, cb("p5"))
t.gate.set()
await done.wait()
await s.stop()
return got
got = run(body)
assert [c[1] for c in t.calls] == [[10], [30, 40], [20, 50]]
assert [c[0].beam for c in t.calls] == [1, 2, 1]
assert got == [("p-first", "n10"), ("f3", "n30"), ("f4", "n40"), ("p2", "n20"), ("p5", "n50")]
@pytest.mark.unit
def test_a_newer_partial_replaces_the_waiting_one_for_the_same_speaker():
t = FakeTranscriber(hold_first=True)
async def body():
s = SttScheduler(t)
s.start()
got = []
done = asyncio.Event()
s.submit_final(("a", "x"), clip(5), FINAL, lambda r: None)
await _wait_entered(t)
s.submit_partial(("a", "1"), clip(10), PARTIAL, lambda r: got.append(r.text))
s.submit_partial(("a", "1"), clip(20), PARTIAL, lambda r: (got.append(r.text), done.set()))
t.gate.set()
await done.wait()
await asyncio.sleep(0.05)
await s.stop()
return got
assert run(body) == ["n20"]
assert [c[1] for c in t.calls] == [[5], [20]]
@pytest.mark.unit
def test_dropped_partials_are_never_transcribed():
t = FakeTranscriber(hold_first=True)
async def body():
s = SttScheduler(t)
s.start()
done = asyncio.Event()
got = []
s.submit_final(("a", "x"), clip(5), FINAL, lambda r: None)
await _wait_entered(t)
s.submit_partial(("a", "1"), clip(10), PARTIAL, lambda r: got.append(r.text))
s.submit_partial(("b", "1"), clip(20), PARTIAL, lambda r: (got.append(r.text), done.set()))
s.drop_partials(lambda key: key[0] == "a")
t.gate.set()
await done.wait()
await s.stop()
return got
assert run(body) == ["n20"]
@pytest.mark.unit
def test_different_decode_settings_are_not_mixed_in_one_batch():
t = FakeTranscriber(hold_first=True)
other = DecodeOptions(beam=2, language=None, hotwords=("자비스",))
async def body():
s = SttScheduler(t)
s.start()
n = []
done = asyncio.Event()
def cb(r):
n.append(r)
if len(n) == 4:
done.set()
s.submit_final(("a", "x"), clip(1), FINAL, cb)
await _wait_entered(t)
s.submit_final(("a", "1"), clip(10), FINAL, cb)
s.submit_final(("b", "1"), clip(20), other, cb)
s.submit_final(("a", "2"), clip(30), FINAL, cb)
t.gate.set()
await done.wait()
await s.stop()
run(body)
assert sorted(c[1] for c in t.calls[1:]) == [[10, 30], [20]]
@pytest.mark.unit
def test_batches_are_capped():
t = FakeTranscriber(hold_first=True)
async def body():
s = SttScheduler(t, max_batch=3)
s.start()
n = []
done = asyncio.Event()
def cb(r):
n.append(r)
if len(n) == 8:
done.set()
s.submit_final(("a", "x"), clip(1), FINAL, cb)
await _wait_entered(t)
for i in range(7):
s.submit_final(("a", str(i)), clip(10 + i), FINAL, cb)
t.gate.set()
await done.wait()
await s.stop()
run(body)
assert [len(c[1]) for c in t.calls] == [1, 3, 3, 1]
@pytest.mark.unit
def test_a_failing_batch_reports_the_error_and_the_scheduler_keeps_going():
t = FakeTranscriber(fail_on_beam=2)
async def body():
s = SttScheduler(t)
s.start()
out = []
done = asyncio.Event()
s.submit_final(("a", "1"), clip(10), FINAL, out.append)
s.submit_partial(("a", "2"), clip(20), PARTIAL, lambda r: (out.append(r), done.set()))
await done.wait()
await s.stop()
return out
out = run(body)
assert out[0].text == "" and out[0].error
assert out[1].text == "n20" and out[1].error is None
assert out[0].stt_ms >= 0
class StepTranscriber:
"""Every call waits for one ``step()``, so the test decides when each batch finishes."""
def __init__(self):
self.calls: list[list[int]] = []
self._steps = threading.Semaphore(0)
def step(self):
self._steps.release()
def transcribe(self, clips, opts):
self.calls.append([len(c) for c in clips])
self._steps.acquire(timeout=5)
return ["" for _ in clips]
@pytest.mark.unit
def test_partial_batches_are_kept_small_so_finals_can_cut_in():
t = StepTranscriber()
async def until(n):
while len(t.calls) < n:
await asyncio.sleep(0.005)
async def body():
s = SttScheduler(t, max_partial_batch=2)
s.start()
for i in range(5):
s.submit_partial(("a", str(i)), clip(10 + i), PARTIAL, lambda r: None)
await until(1) # first partial batch is running
s.submit_final(("a", "late"), clip(99), FINAL, lambda r: None)
t.step()
await until(2)
t.step()
await until(3)
t.step()
await until(4)
t.step()
await asyncio.sleep(0.02)
await s.stop()
run(body)
assert t.calls == [[10, 11], [99], [12, 13], [14]]

View File

@@ -0,0 +1,191 @@
"""Voice worker: per-speaker utterance segmentation (start, partial cadence, the four end reasons).
The fake VAD reads the speech probability straight off the audio: a window of constant amplitude
0.9 is "speech at 0.9". That keeps the tests about the segmentation rules, not about Silero.
"""
from __future__ import annotations
import numpy as np
import pytest
from voice_worker.segmenter import Final, Partial, Segmenter, SpeechStart
from voice_worker.settings import SttSettings
SR = 16000
FRAME_MS = 20
class AmplitudeVad:
def prob(self, window: np.ndarray) -> float:
return float(np.max(np.abs(window)))
class Feeder:
def __init__(self, settings: SttSettings | None = None):
self.settings = settings or SttSettings()
self.seg = Segmenter(AmplitudeVad(), self.settings)
self.now = 100.0
self.events: list = []
def feed(self, level: float, ms: int) -> list:
got = []
for _ in range(ms // FRAME_MS):
self.now += FRAME_MS / 1000
got += self.seg.push(np.full(SR * FRAME_MS // 1000, level, dtype=np.float32), self.now)
self.events += got
return got
def of(self, cls) -> list:
return [e for e in self.events if isinstance(e, cls)]
@pytest.mark.unit
def test_silence_produces_nothing():
f = Feeder()
f.feed(0.0, 3000)
assert f.events == []
@pytest.mark.unit
def test_speech_then_silence_ends_with_one_final():
f = Feeder()
f.feed(0.0, 500)
f.feed(0.9, 1000)
f.feed(0.0, 500)
starts, finals = f.of(SpeechStart), f.of(Final)
assert [s.utt for s in starts] == [1]
assert len(finals) == 1
fin = finals[0]
assert fin.utt == 1 and fin.reason == "silence"
# Pre-roll (<= 200 ms) + 1 s speech + trailing silence capped at 200 ms.
assert 1.0 <= len(fin.audio) / SR <= 1.45
assert fin.audio.dtype == np.float32
@pytest.mark.unit
def test_final_waits_for_the_configured_silence():
f = Feeder(SttSettings(end_silence_ms=600))
f.feed(0.9, 800)
f.feed(0.0, 400)
assert f.of(Final) == []
f.feed(0.0, 300)
assert len(f.of(Final)) == 1
@pytest.mark.unit
def test_changed_settings_apply_to_the_running_utterance():
f = Feeder(SttSettings(end_silence_ms=1500))
f.feed(0.9, 800)
f.settings.update({"end_silence_ms": 200})
f.feed(0.0, 300)
assert [e.reason for e in f.of(Final)] == ["silence"]
@pytest.mark.unit
def test_a_short_blip_is_dropped_without_any_message():
f = Feeder()
f.feed(0.9, 120)
f.feed(0.0, 1000)
assert f.events == []
@pytest.mark.unit
def test_hysteresis_keeps_speech_going_but_never_starts_it():
f = Feeder()
f.feed(0.4, 1000)
assert f.events == []
f.feed(0.9, 400)
f.feed(0.4, 1000) # below the start threshold but above the keep threshold
assert f.of(Final) == []
f.feed(0.0, 400)
assert len(f.of(Final)) == 1
@pytest.mark.unit
def test_speaking_end_closes_the_utterance_at_once():
f = Feeder()
f.feed(0.9, 700)
events = f.seg.speaking_end(f.now)
assert [type(e) for e in events] == [Final]
assert events[0].reason == "speaking_end"
assert events[0].decided_at == f.now
@pytest.mark.unit
def test_speaking_end_after_a_blip_sends_nothing():
f = Feeder()
f.feed(0.9, 100)
assert f.seg.speaking_end(f.now) == []
@pytest.mark.unit
def test_audio_that_stops_arriving_ends_as_idle():
f = Feeder()
f.feed(0.9, 700)
assert f.seg.tick(f.now + 0.2) == []
events = f.seg.tick(f.now + 0.31)
assert [e.reason for e in events] == ["idle"]
assert f.seg.tick(f.now + 1.0) == []
@pytest.mark.unit
def test_long_speech_is_cut_at_the_whisper_window_and_continues_as_a_new_utterance():
f = Feeder()
f.feed(0.9, 29_000)
finals = f.of(Final)
assert [e.reason for e in finals] == ["max_length"]
assert len(finals[0].audio) / SR <= 28.0
assert [s.utt for s in f.of(SpeechStart)] == [1, 2]
@pytest.mark.unit
def test_partials_follow_the_interval_and_grow():
f = Feeder(SttSettings(partial_interval_ms=400))
f.feed(0.9, 2000)
partials = f.of(Partial)
assert 3 <= len(partials) <= 5
lengths = [len(p.audio) for p in partials]
assert lengths == sorted(lengths) and len(set(lengths)) == len(lengths)
assert all(p.utt == 1 for p in partials)
f.feed(0.0, 400)
n = len(f.of(Partial))
f.feed(0.0, 1000)
assert len(f.of(Partial)) == n # nothing after the final
@pytest.mark.unit
def test_no_partials_before_speech_start():
f = Feeder(SttSettings(partial_interval_ms=100))
f.feed(0.9, 200)
assert f.of(Partial) == []
@pytest.mark.unit
def test_utterance_numbers_count_up_per_speaker():
f = Feeder()
for _ in range(3):
f.feed(0.9, 500)
f.feed(0.0, 400)
assert [e.utt for e in f.of(Final)] == [1, 2, 3]
@pytest.mark.unit
@pytest.mark.parametrize("bad", [{"end_silence_ms": 50}, {"end_silence_ms": 5000}, {"final_beam": 0},
{"final_beam": 9}, {"partials": "yes"}, {"hotwords": "자비스"},
{"language": 3}, {"partial_interval_ms": 10}, {"nope": 1}])
def test_bad_settings_are_refused_and_change_nothing(bad):
s = SttSettings()
before = s.as_dict()
with pytest.raises(ValueError):
s.update(bad)
assert s.as_dict() == before
@pytest.mark.unit
def test_settings_accept_null_language_and_hotword_lists():
s = SttSettings()
s.update({"language": None, "hotwords": ["자비스", "Jarvis"], "partials": False})
assert s.as_dict()["language"] is None
assert s.hotwords == ("자비스", "Jarvis")
assert s.partials is False

View File

@@ -0,0 +1,50 @@
"""Voice worker: sentence splitting for streaming TTS, whole and incremental."""
from __future__ import annotations
import pytest
from voice_worker.sentences import StreamingSplitter, split_sentences
@pytest.mark.unit
def test_splits_on_terminal_punctuation_only():
assert split_sentences("안녕하세요. 오늘은 맑아요! 우산은 필요 없어요?") == [
"안녕하세요.", "오늘은 맑아요!", "우산은 필요 없어요?"]
@pytest.mark.unit
def test_decimals_versions_and_urls_stay_whole():
assert split_sentences("기온은 17.5도예요. 자세한 건 example.com 에 있어요.") == [
"기온은 17.5도예요.", "자세한 건 example.com 에 있어요."]
@pytest.mark.unit
def test_cjk_terminators_split_without_spaces():
assert split_sentences("今日は晴れです。明日は雨です。") == ["今日は晴れです。", "明日は雨です。"]
@pytest.mark.unit
def test_short_fragments_merge_into_a_neighbour():
assert split_sentences("네. 알람을 일곱 시에 맞췄어요.") == ["네. 알람을 일곱 시에 맞췄어요."]
@pytest.mark.unit
def test_streaming_holds_the_unfinished_tail_until_more_text_arrives():
s = StreamingSplitter()
assert s.feed("오늘 서울은 맑") == []
assert s.feed("아요. 기온은 17") == ["오늘 서울은 맑아요."]
assert s.feed(".5도예요. 바람은") == ["기온은 17.5도예요."]
assert s.finish() == ["바람은"]
@pytest.mark.unit
def test_streaming_output_equals_whole_text_split():
text = "첫 문장입니다. 두 번째 문장이에요! 세 번째는 질문인가요? 마지막 문장이에요."
s = StreamingSplitter()
got = []
for i in range(0, len(text), 3):
got += s.feed(text[i:i + 3])
got += s.finish()
assert got == split_sentences(text)

View File

@@ -0,0 +1,327 @@
"""Voice worker end to end over a real local WebSocket, with fake STT/VAD/TTS engines.
The fake VAD reads speech probability off the amplitude and the fake transcriber names a clip by its
peak level, so each speaker's text tells which speaker's audio reached the model.
"""
from __future__ import annotations
import asyncio
import json
import time
import numpy as np
import pytest
from voice_worker.protocol import decode_tts_frame, encode_audio_frame
from voice_worker.server import CLOSE_BAD_HELLO, CLOSE_BAD_TOKEN, VoiceWorker
websockets = pytest.importorskip("websockets")
from websockets.asyncio.client import connect # noqa: E402
TOKEN = "test-token"
class AmplitudeVad:
def prob(self, window):
return float(np.max(np.abs(window)))
class LevelTranscriber:
def __init__(self):
self.calls = []
def transcribe(self, clips, opts):
self.calls.append((opts, len(clips)))
return [f"level {np.median(np.abs(c)):.1f}" for c in clips]
class FakeTts:
"""10 samples per character; optional delay per sentence; can fail on a marker."""
def __init__(self, delay: float = 0.0):
self.delay = delay
self.texts = []
def synthesize(self, text, rate):
if self.delay:
time.sleep(self.delay)
if "FAIL" in text:
raise RuntimeError("engine broke")
self.texts.append(text)
return np.full(len(text) * 10, 1000, dtype=np.int16)
def pcm(level: float, ms: int, rate=16000, channels=1) -> bytes:
n = rate * ms // 1000
return (np.full(n * channels, level) * 32767).astype("<i2").tobytes()
async def with_worker(body, transcriber=None, tts=None, **kw):
worker = VoiceWorker(TOKEN, transcriber if transcriber is not None else LevelTranscriber(), AmplitudeVad,
tts, **kw)
server = await worker.serve("127.0.0.1", 0)
port = server.sockets[0].getsockname()[1]
try:
return await asyncio.wait_for(body(f"ws://127.0.0.1:{port}"), 15)
finally:
server.close()
await server.wait_closed()
await worker.stop()
async def hello(ws, **extra):
await ws.send(json.dumps({"type": "hello", "token": TOKEN, "audio": {"rate": 16000, "channels": 1},
"tts": {"rate": 24000, "channels": 2}, **extra}))
return json.loads(await ws.recv())
async def recv_until(ws, pred, limit=200):
seen = []
for _ in range(limit):
raw = await ws.recv()
msg = raw if isinstance(raw, bytes) else json.loads(raw)
seen.append(msg)
if pred(msg):
return seen
raise AssertionError("condition never met")
async def stream(ws, speaker: str, level: float, ms: int, realtime: bool = False):
for _ in range(ms // 20):
await ws.send(encode_audio_frame(speaker, pcm(level, 20)))
if realtime:
await asyncio.sleep(0.02)
@pytest.mark.unit
def test_wrong_token_is_refused():
async def body(url):
async with connect(url) as ws:
await ws.send(json.dumps({"type": "hello", "token": "nope"}))
with pytest.raises(websockets.ConnectionClosed) as e:
await ws.recv()
return e.value.rcvd.code
assert asyncio.run(with_worker(body)) == CLOSE_BAD_TOKEN
@pytest.mark.unit
def test_silent_client_is_dropped_after_the_hello_timeout():
async def body(url):
async with connect(url) as ws:
with pytest.raises(websockets.ConnectionClosed) as e:
await ws.recv()
return e.value.rcvd.code
assert asyncio.run(with_worker(body, hello_timeout_s=0.2)) == CLOSE_BAD_HELLO
@pytest.mark.unit
def test_non_loopback_addresses_are_refused():
worker = VoiceWorker(TOKEN, LevelTranscriber(), AmplitudeVad)
with pytest.raises(ValueError):
asyncio.run(worker.serve("0.0.0.0", 0))
@pytest.mark.unit
def test_hello_ok_reports_readiness_and_effective_settings():
async def body(url):
async with connect(url) as ws:
return await hello(ws, stt={"hotwords": ["자비스"], "end_silence_ms": 250})
ok = asyncio.run(with_worker(body))
assert ok["type"] == "hello_ok"
assert ok["stt_ready"] is True and ok["tts_ready"] is False
assert ok["stt"]["hotwords"] == ["자비스"] and ok["stt"]["end_silence_ms"] == 250
assert ok["tts"] == {"rate": 24000, "channels": 2}
@pytest.mark.unit
def test_two_speakers_each_get_their_own_final():
async def body(url):
async with connect(url) as ws:
await hello(ws, stt={"partials": False})
await stream(ws, "alice", 0.9, 600)
await stream(ws, "bob", 0.6, 600)
await ws.send(json.dumps({"type": "speaking_end", "speaker": "alice"}))
await ws.send(json.dumps({"type": "speaking_end", "speaker": "bob"}))
seen = await recv_until(ws, lambda m: m.get("type") == "final" and m["speaker"] == "bob")
if not any(m.get("type") == "final" and m["speaker"] == "alice" for m in seen):
seen += await recv_until(ws, lambda m: m.get("type") == "final")
return seen
seen = asyncio.run(with_worker(body))
finals = {m["speaker"]: m for m in seen if m.get("type") == "final"}
assert finals["alice"]["text"] == "level 0.9" and finals["bob"]["text"] == "level 0.6"
assert {m["end_reason"] for m in finals.values()} == {"speaking_end"}
assert all(m["utt"] == 1 and 500 <= m["audio_ms"] <= 900 for m in finals.values())
starts = [m for m in seen if m.get("type") == "speech_start"]
assert sorted(m["speaker"] for m in starts) == ["alice", "bob"]
assert not any(m.get("type") == "partial" for m in seen)
@pytest.mark.unit
def test_audio_that_stops_without_speaking_end_still_ends():
async def body(url):
async with connect(url) as ws:
await hello(ws, stt={"partials": False, "end_silence_ms": 200})
await stream(ws, "alice", 0.9, 500)
return await recv_until(ws, lambda m: m.get("type") == "final")
final = asyncio.run(with_worker(body))[-1]
assert final["end_reason"] == "idle" and final["text"] == "level 0.9"
@pytest.mark.unit
def test_partials_flow_while_speaking_and_stop_when_turned_off():
async def body(url):
async with connect(url) as ws:
await hello(ws, stt={"partial_interval_ms": 100})
await stream(ws, "alice", 0.9, 800, realtime=True)
first = await recv_until(ws, lambda m: m.get("type") == "partial")
await ws.send(json.dumps({"type": "config", "partials": False}))
# Drain whatever was already on its way before the setting changed.
while True:
try:
await asyncio.wait_for(ws.recv(), 0.3)
except asyncio.TimeoutError:
break
await stream(ws, "alice", 0.9, 600, realtime=True)
await ws.send(json.dumps({"type": "speaking_end", "speaker": "alice"}))
rest = await recv_until(ws, lambda m: m.get("type") == "final")
return first, rest
first, rest = asyncio.run(with_worker(body))
assert first[-1]["text"] == "level 0.9" and first[-1]["utt"] == 1
assert not any(m.get("type") == "partial" for m in rest)
assert rest[-1]["type"] == "final"
@pytest.mark.unit
def test_tts_streams_one_chunk_per_sentence_in_the_session_format():
tts = FakeTts()
async def body(url):
async with connect(url) as ws:
await hello(ws)
await ws.send(json.dumps({"type": "tts", "id": "r1", "text": "오늘은 맑아요. 기온은 17.5도예요."}))
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
seen = asyncio.run(with_worker(body, tts=tts))
chunks = [decode_tts_frame(m) for m in seen if isinstance(m, bytes)]
assert [(c[0], c[1]) for c in chunks] == [("r1", 0), ("r1", 1)]
assert tts.texts == ["오늘은 맑아요.", "기온은 17.5도예요."]
# Stereo: every mono sample twice, 16-bit.
assert len(chunks[0][2]) == len("오늘은 맑아요.") * 10 * 2 * 2
assert seen[-1] == {"type": "tts_done", "id": "r1", "chunks": 2, "cancelled": False, "failed": False}
@pytest.mark.unit
def test_tts_text_can_arrive_in_pieces():
tts = FakeTts()
async def body(url):
async with connect(url) as ws:
await hello(ws)
for piece, more in [("첫 번째 문장", True), ("이에요. 두 번", True), ("째 문장이에요.", False)]:
await ws.send(json.dumps({"type": "tts", "id": "s", "text": piece, "more": more}))
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
seen = asyncio.run(with_worker(body, tts=tts))
assert tts.texts == ["첫 번째 문장이에요.", "두 번째 문장이에요."]
assert seen[-1]["chunks"] == 2
@pytest.mark.unit
def test_cancel_stops_the_remaining_sentences():
tts = FakeTts(delay=0.15)
async def body(url):
async with connect(url) as ws:
await hello(ws)
text = " ".join(f"이것은 {i}번째 문장입니다." for i in range(6))
await ws.send(json.dumps({"type": "tts", "id": "c", "text": text}))
first = await recv_until(ws, lambda m: isinstance(m, bytes))
await ws.send(json.dumps({"type": "tts_cancel", "id": "c"}))
return first + await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
seen = asyncio.run(with_worker(body, tts=tts))
done = seen[-1]
assert done["cancelled"] is True and done["chunks"] < 6
assert sum(isinstance(m, bytes) for m in seen) == done["chunks"]
@pytest.mark.unit
def test_a_failing_sentence_ends_the_request_as_failed():
tts = FakeTts()
async def body(url):
async with connect(url) as ws:
await hello(ws)
await ws.send(json.dumps({"type": "tts", "id": "f", "text": "첫 문장입니다. FAIL 문장입니다. 셋째 문장입니다."}))
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done")
seen = asyncio.run(with_worker(body, tts=tts))
errors = [m for m in seen if isinstance(m, dict) and m.get("type") == "error"]
assert errors and errors[0]["code"] == "tts_failed" and errors[0]["id"] == "f"
assert seen[-1]["failed"] is True and seen[-1]["chunks"] == 1
assert tts.texts == ["첫 문장입니다."]
@pytest.mark.unit
def test_tts_without_an_engine_reports_unavailable():
async def body(url):
async with connect(url) as ws:
await hello(ws)
await ws.send(json.dumps({"type": "tts", "id": "x", "text": "안녕하세요."}))
return json.loads(await ws.recv())
msg = asyncio.run(with_worker(body))
assert msg["type"] == "error" and msg["code"] == "tts_unavailable" and msg["id"] == "x"
@pytest.mark.unit
def test_bad_messages_get_an_error_and_the_connection_stays_open():
async def body(url):
async with connect(url) as ws:
await hello(ws)
out = []
for raw in ["{not json", json.dumps({"type": "dance"}), json.dumps({"type": "config", "final_beam": 99})]:
await ws.send(raw)
out.append(json.loads(await ws.recv()))
await ws.send(b"\x01\x00")
out.append(json.loads(await ws.recv()))
await ws.send(json.dumps({"type": "config", "final_beam": 3}))
await ws.send(json.dumps({"type": "dance"}))
out.append(json.loads(await ws.recv()))
return out
out = asyncio.run(with_worker(body))
assert [m["code"] for m in out] == ["bad_message", "bad_message", "bad_message", "bad_frame", "bad_message"]
@pytest.mark.unit
def test_48k_stereo_input_reaches_the_model_at_16k():
t = LevelTranscriber()
lengths = []
orig = t.transcribe
def spy(clips, opts):
lengths.extend(len(c) for c in clips)
return orig(clips, opts)
t.transcribe = spy
async def body(url):
async with connect(url) as ws:
await ws.send(json.dumps({"type": "hello", "token": TOKEN, "stt": {"partials": False}}))
await ws.recv()
for _ in range(30): # 600 ms
await ws.send(encode_audio_frame("a", pcm(0.9, 20, rate=48000, channels=2)))
await ws.send(json.dumps({"type": "speaking_end", "speaker": "a"}))
return await recv_until(ws, lambda m: m.get("type") == "final")
final = asyncio.run(with_worker(body, transcriber=t))[-1]
assert final["text"] == "level 0.9"
assert 0.5 * 16000 <= lengths[0] <= 0.9 * 16000

View File

@@ -0,0 +1,53 @@
"""Voice worker: the streaming Silero VAD (real ONNX model bundled with faster-whisper, CPU only)."""
from __future__ import annotations
import wave
from pathlib import Path
import numpy as np
import pytest
pytest.importorskip("onnxruntime")
pytest.importorskip("faster_whisper")
from voice_worker.segmenter import WINDOW # noqa: E402
from voice_worker.vad import SileroVad # noqa: E402
SAMPLE = Path(__file__).resolve().parent.parent / "scripts" / "m0" / "assets" / "ko_samples" / "ko_01.wav"
def _speech() -> np.ndarray:
with wave.open(str(SAMPLE), "rb") as w:
return np.frombuffer(w.readframes(w.getnframes()), dtype="<i2").astype(np.float32) / 32768.0
def _probs(stream, audio):
n = len(audio) // WINDOW * WINDOW
return np.array([stream.prob(audio[i:i + WINDOW]) for i in range(0, n, WINDOW)])
@pytest.fixture(scope="module")
def vad():
return SileroVad()
@pytest.mark.unit
def test_speech_is_detected_and_silence_is_not(vad):
speech = _probs(vad.stream(), _speech())
silence = _probs(vad.stream(), np.zeros(16000, dtype=np.float32))
assert (speech >= 0.5).mean() > 0.4
assert silence.max() < 0.2
@pytest.mark.unit
def test_each_speaker_stream_keeps_its_own_state(vad):
audio = _speech()
alone = _probs(vad.stream(), audio)
a, b = vad.stream(), vad.stream()
n = len(audio) // WINDOW * WINDOW
interleaved = []
for i in range(0, n, WINDOW):
interleaved.append(a.prob(audio[i:i + WINDOW]))
b.prob(np.zeros(WINDOW, dtype=np.float32)) # another speaker in silence between every window
assert np.allclose(alone, interleaved, atol=1e-5)