fix(voice_worker): even out speaker loudness and stop transcribing noise into words

- Loudness equalisation replaces peak normalisation: rumble below 60 Hz removed, level measured on
  the VAD-voiced windows only, brought to -20 dBFS (gain -20..+40 dB), soft limiter below 0.98.
- Only clearly voiced windows (VAD >= 0.5) count towards the 250 ms needed before an utterance
  reaches the model, so noise the VAD half-believes never calls Whisper.
- Hallucination gate: mean token log-probability >= -0.7 (the old len/(len+1) correction let short
  hallucinations through), compression ratio <= 2.4; no_speech_prob dropped (always 0.0 on turbo).
- With hotwords, the plain decode of the same encoder pass decides; the hint may fix a wake word's
  spelling but cannot add or drop one (it wrote the wake word on laughter and after real sentences).
- Opt-in real-model evals: voice-like babble with the hint, speech over babble, quiet and clipped
  speakers, and a JARVIS_NOISE_DIR folder of real noise recordings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-11 10:27:38 +09:00
parent a7685e9078
commit 590cd07268
11 changed files with 460 additions and 54 deletions

View File

@@ -22,7 +22,9 @@
- **2단 디코딩**: 부분 전사는 beam 1(빠름), 최종 전사는 beam 2~3(정확도)입니다.
- **게임 중 LLM이 답하는 동안**: STT와 LLM은 같은 GPU를 씁니다. 10명이 쉬지 않고 말하며 부분 전사를 계속 돌리면 LLM 출력이 1/2.5~1/3로 줄었습니다(M0 `gamemode_stt`). 그래서 게임 중에 LLM이 답을 만드는 동안에는 부분 전사를 멈추고(그동안 실시간 자막이 바뀌지 않음), 말이 끝난 뒤의 최종 전사는 기다리지 않고 바로 처리합니다. 게임이 없을 때는 LLM이 답하는 동안에도 부분 전사를 그대로 합니다(사용자 결정, 2026-11-10). 최종 전사가 부분 전사보다 먼저입니다. 멈추기로 한 것은 M0 `gamemode_tune` 측정 결과입니다(2026-11-11): 2초 간격은 37.5 tok/s, 음성 첫 문장 0.89초에 최종 전사 대기가 최대 1.2초였고, 멈추면 64.9 tok/s, 0.55초였습니다. 부분 전사를 멈췄을 때 beam 2~3 최종 전사와 긴 말의 실제 지연은 아직 재지 않았습니다([pc-features.md](pc-features.md#자동-조절-측정-m0-gamemode_tune-2026-11-11)).
- **부분 전사는 2명씩만 묶음(M1 실측)**: Whisper 인코더는 말 길이와 상관없이 늘 30초 창을 계산해서, 묶어도 비용이 사람 수에 비례합니다(RTX 5050에서 1명 0.2초, 10명 2.1초). 10명분 부분 전사를 한 번에 돌리면 그동안 들어온 최종 전사가 2초 넘게 기다려서, 부분 전사는 한 번에 2명까지만 돌리고 그 사이에 최종 전사가 먼저 들어가게 했습니다. 여러 명이 함께 말하면 자막은 늦어지고 사람마다 가장 최근 소리만 전사합니다. 인코더 창을 줄이면 4~7배 빨라졌지만 같은 문장을 되풀이하는 오류가 생겨 쓰지 않습니다.
- **호출어 힌트**: 등록한 호출어를 STT에 힌트로 넘겨 잘못 알아듣는 일을 줄입니다.
- **호출어 힌트**: 등록한 호출어를 STT에 힌트로 넘겨 잘못 알아듣는 일을 줄입니다. 힌트만 쓰면 웃음소리에서 "자비스"를 지어내거나 실제 문장 끝에 "자비스"를 붙여서(M1 측정, 2026-11-11), 힌트가 있으면 같은 소리를 힌트 없이도 한 번 더 해석해 그 결과로 판단합니다. 힌트는 호출어 철자를 고치는 데만 쓰이고 호출어를 더하거나 빼지 못합니다.
- **음량 맞추기**: 사람마다 목소리 크기가 달라도 같은 크기로 맞춰 STT에 넣습니다. 목소리가 있는 구간만 재서 -20dBFS로 맞추고(최대 +40dB, 낮은 웅웅거림 제거, 큰 소리는 찌그러지지 않게 부드럽게 눌러 줌), 클릭이나 노크 소리가 음량을 정하지 않게 합니다. 측정하면 -60dBFS 속삭임부터 찌그러질 만큼 큰 소리까지 똑같이 받아 적었습니다.
- **잡음 거르기와 헛 전사 막기**: 목소리가 분명한 구간(음성 확률 0.5 이상)이 0.25초 넘게 있어야 STT를 돌립니다. 키보드, 마우스, 발소리, 청소기 같은 잡음은 여기서 걸러집니다. 기침, 웃음처럼 목소리 같은 소리는 STT까지 가지만 결과의 토큰당 평균 확신도(-0.7 이상)와 반복 여부로 거릅니다. 실제 잡음 54개(ESC-50)를 원래 크기와 -25dBFS로 넣은 108개 중 102개는 STT까지 가지 않았고, STT까지 간 6개(기침, 웃음, 울음)도 모두 빈 결과였습니다(호출어 힌트가 있을 때도 같음). 그 잡음 위에 말한 문장 89개는 문자 오류율 평균 4.8%였고, 남은 문제는 웃음 직후 "감사합니다." 한 번(확신도가 기준선에 걸침), 시끄러운 잡음 속 짧은 단어 2개 누락, 같은 단어 세 번 반복 한 번입니다. 이 모델(large-v3-turbo)의 "말 없음" 확률은 늘 0이라 쓰지 않습니다.
- **RealtimeSTT 사용 범위**: RealtimeSTT는 화자 하나에 인스턴스 하나 구조라 10명이면 모델을 10번 올립니다. 그래서 설계(부분/최종 전사, VAD 결합)만 참고하고, **모델 1개를 10명이 공유하는 배치 방식**으로 직접 구현합니다.
- **선택지**: 더 낮은 지연이 필요하면 sherpa-onnx 스트리밍 모델을 "초고속" 옵션으로 검토합니다(한국어 정확도 확인 후).

View File

@@ -50,9 +50,39 @@ class InputConverter:
return out.astype(np.float32)
def peak_normalise(x: np.ndarray, target: float = 0.9, max_gain: float = 10.0) -> np.ndarray:
peak = float(np.max(np.abs(x))) if len(x) else 0.0
if peak <= 0:
return x.astype(np.float32)
gain = min(target / peak, max_gain)
return (x * gain).astype(np.float32)
TARGET_DB = -20.0 # loudness every speaker is brought to, dBFS RMS over the voiced windows
MAX_GAIN_DB = 40.0 # a whisper at -60 dBFS still reaches the target; pure hiss is not blown up
MIN_GAIN_DB = -20.0
HIGHPASS_HZ = 60.0 # mains hum and desk rumble carry no speech
_KNEE = 0.8 # soft limiter: peaks above this bend smoothly towards the ceiling instead of clipping
_CEILING = 0.98
_WINDOW = 512
def loudness_equalise(x: np.ndarray, voiced: np.ndarray | None = None) -> np.ndarray:
"""Bring a clip to the same loudness whoever spoke it, without clipping.
``voiced`` marks the 512-sample windows the VAD heard as speech; the level is measured on them so
silence, a click or a knock does not decide the gain. Without it the louder half of the windows is
used. Low rumble is removed first and peaks are soft-limited after the gain.
"""
x = np.asarray(x, dtype=np.float32)
if len(x) == 0:
return x
spec = np.fft.rfft(x.astype(np.float64))
spec[np.fft.rfftfreq(len(x), 1 / TARGET_RATE) < HIGHPASS_HZ] = 0
y = np.fft.irfft(spec, len(x))
n_win = -(-len(y) // _WINDOW)
frames = np.pad(y, (0, n_win * _WINDOW - len(y))).reshape(n_win, _WINDOW)
energy = (frames ** 2).mean(axis=1)
if voiced is not None and len(voiced) == n_win and voiced.any():
sel = energy[voiced]
else:
sel = energy[energy >= np.median(energy)]
level_db = 10 * np.log10(sel.mean() + 1e-20)
gain_db = float(np.clip(TARGET_DB - level_db, MIN_GAIN_DB, MAX_GAIN_DB))
y = y * 10 ** (gain_db / 20)
over = np.abs(y) > _KNEE
room = _CEILING - _KNEE
y[over] = np.sign(y[over]) * (_KNEE + room * np.tanh((np.abs(y[over]) - _KNEE) / room))
return y.astype(np.float32)

View File

@@ -33,6 +33,7 @@ class SpeechStart:
class Partial:
utt: int
audio: np.ndarray
voiced: np.ndarray # one bool per 512-sample window: the VAD heard speech there
@dataclass
@@ -41,6 +42,7 @@ class Final:
audio: np.ndarray
reason: str # silence | speaking_end | idle | max_length
decided_at: float
voiced: np.ndarray
def _windows(ms: float) -> int:
@@ -60,7 +62,8 @@ class Segmenter:
def _reset(self) -> None:
self._active = False
self._win: list[np.ndarray] = []
self._speech = 0
self._voiced: list[bool] = []
self._speech = 0 # windows that clearly held speech (>= START_PROB)
self._trailing = 0
self._started = False
self._last_partial_at = 0.0
@@ -83,7 +86,7 @@ class Segmenter:
and now - self._last_partial_at >= self.settings.partial_interval_ms / 1000:
self._last_partial_at = now
self._last_partial_len = len(self._win)
events.append(Partial(self._utt_count, np.concatenate(self._win)))
events.append(Partial(self._utt_count, np.concatenate(self._win), np.array(self._voiced)))
return events
def _window(self, w: np.ndarray, now: float) -> list:
@@ -92,6 +95,7 @@ class Segmenter:
if p >= START_PROB:
self._active = True
self._win = self._pre + [w]
self._voiced = [False] * len(self._pre) + [True]
self._pre = []
self._speech = 1
self._trailing = 0
@@ -100,8 +104,12 @@ class Segmenter:
del self._pre[:-_windows(PRE_ROLL_MS)]
return []
self._win.append(w)
if p >= KEEP_PROB:
self._voiced.append(p >= START_PROB)
# "Maybe speech" (between the two thresholds) keeps an utterance going but is not evidence
# of a voice, so noise the VAD half-believes never adds up to a call to the model.
if p >= START_PROB:
self._speech += 1
if p >= KEEP_PROB:
self._trailing = 0
else:
self._trailing += 1
@@ -121,8 +129,9 @@ class Segmenter:
events: list = []
if self._active and self._started:
drop = max(0, self._trailing - _windows(TAIL_KEEP_MS))
kept = self._win[:len(self._win) - drop]
events.append(Final(self._utt_count, np.concatenate(kept), reason, now))
keep = len(self._win) - drop
events.append(Final(self._utt_count, np.concatenate(self._win[:keep]), reason, now,
np.array(self._voiced[:keep])))
self._reset()
return events

View File

@@ -15,7 +15,7 @@ import numpy as np
from jarvis.debug import debug_log
from .audio import InputConverter
from .audio import InputConverter, loudness_equalise
from .protocol import KIND_AUDIO, ProtocolError, decode_audio_frame, encode_tts_frame
from .scheduler import DecodeOptions, SttResult, SttScheduler, Transcriber
from .segmenter import SR, Final, Partial, Segmenter, SpeechStart, VadStream
@@ -211,7 +211,7 @@ class Session:
self.send({"type": "speech_start", "speaker": speaker, "utt": ev.utt})
elif isinstance(ev, Partial):
if sched and self.settings.partials:
sched.submit_partial((self.sid, speaker), ev.audio, self._opts(1),
sched.submit_partial((self.sid, speaker), loudness_equalise(ev.audio, ev.voiced), self._opts(1),
self._partial_cb(speaker, st, ev.utt))
elif isinstance(ev, Final):
debug_log(f"utterance end speaker={speaker} utt={ev.utt} reason={ev.reason} "
@@ -219,7 +219,8 @@ class Session:
if sched is None:
self._send_final(speaker, ev, SttResult("", 0, "stt unavailable"))
else:
sched.submit_final((self.sid, speaker), ev.audio, self._opts(self.settings.final_beam),
sched.submit_final((self.sid, speaker), loudness_equalise(ev.audio, ev.voiced),
self._opts(self.settings.final_beam),
lambda res, ev=ev: self._send_final(speaker, ev, res))
def _partial_cb(self, speaker: str, st: _Speaker, utt: int):

View File

@@ -4,18 +4,83 @@ from __future__ import annotations
import glob
import os
import re
import sys
import zlib
import numpy as np
from jarvis.debug import debug_log
from .audio import peak_normalise
from .scheduler import DecodeOptions
SR = 16000
NO_SPEECH_THRESHOLD = 0.5 # same gate as v1 (whisper_no_speech_threshold)
MIN_CONFIDENCE = 0.3 # avg_logprob + 1, same as v1 (whisper_min_confidence)
# Mean log-probability per output token. Measured on real noise (coughs, laughter, crying) and speech
# over it: every hallucination stayed below, real speech above except two short words in loud noise.
# Whisper's no_speech_prob is not used: large-v3-turbo reports 0.0 for silence and noise alike.
MIN_MEAN_LOGPROB = -0.7
MAX_COMPRESSION_RATIO = 2.4 # repetition loops ("A B A B A B ...") compress far better than speech
def _letters(text: str) -> str:
"""Text without spacing, punctuation and case, so decodes compare by what was said."""
return re.sub(r"[\W_]+", "", text).casefold()
def compression_ratio(text: str) -> float:
raw = text.encode("utf-8")
return len(raw) / len(zlib.compress(raw)) if raw else 0.0
def keep_transcript(text: str, mean_logprob: float) -> bool:
return bool(_letters(text)) and mean_logprob >= MIN_MEAN_LOGPROB \
and compression_ratio(text) <= MAX_COMPRESSION_RATIO
def _edit_distance(a: str, b: str) -> int:
prev = list(range(len(b) + 1))
for i, ca in enumerate(a, 1):
cur = [i]
for j, cb in enumerate(b, 1):
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
prev = cur
return prev[-1]
def _mentions(text: str, word: str) -> int:
"""How many times ``word`` occurs in ``text``, allowing a near miss in spelling (자비수 for 자비스)."""
t, w = _letters(text), _letters(word)
if not w:
return 0
slack = max(1, len(w) // 3)
count, i = 0, 0
while i <= len(t) - max(1, len(w) - slack):
hit = next((n for n in range(len(w) - slack, len(w) + slack + 1)
if i + n <= len(t) and _edit_distance(t[i:i + n], w) <= slack), None)
if hit:
count += 1
i += hit
else:
i += 1
return count
def choose_with_hint(hint_text: str, hint_lp: float, plain_text: str, plain_lp: float,
hotwords: tuple[str, ...]) -> str:
"""Pick between the decode that had the hotwords as a hint and the plain decode of the same audio.
The hint can make Whisper write a wake word that nobody said (on laughter, or tacked onto a real
sentence), so the plain decode decides: no plain text means nothing was said, and the hint's text
is used only when it stays close to the plain one and mentions every hotword as often (it may fix
their spelling, not add or drop them).
"""
if not keep_transcript(plain_text, plain_lp):
return ""
close = _edit_distance(_letters(hint_text), _letters(plain_text)) <= len(_letters(plain_text)) // 2
if close and keep_transcript(hint_text, hint_lp) and all(
_mentions(hint_text, w) == _mentions(plain_text, w) for w in hotwords):
return hint_text
return plain_text
def add_cuda_dll_dirs() -> None:
@@ -50,11 +115,12 @@ class WhisperTranscriber:
self.transcribe([np.zeros(SR, dtype=np.float32)], DecodeOptions(beam=1, language="en", hotwords=()))
def transcribe(self, clips: list[np.ndarray], opts: DecodeOptions) -> list[str]:
"""Clips arrive loudness-equalised by the session."""
m = self.model
n_frames = m.feature_extractor.nb_max_frames
feats = []
for c in clips:
a = peak_normalise(c[: SR * 30])
a = np.asarray(c[: SR * 30], dtype=np.float32)
a = np.pad(a, (0, SR * 30 - len(a)))
feats.append(m.feature_extractor(a, padding=0)[:, :n_frames])
enc = m.encode(np.stack(feats).astype(np.float32))
@@ -62,17 +128,18 @@ class WhisperTranscriber:
langs = [opts.language] * len(clips)
else:
langs = [r[0][0][2:-2] for r in m.model.detect_language(enc)]
hot = " ".join(opts.hotwords) or None
prompts = [m.get_prompt(self._tokenizer(lang), [], without_timestamps=True, hotwords=hot) for lang in langs]
results = m.model.generate(enc, prompts, beam_size=opts.beam, max_length=224,
suppress_blank=True, suppress_tokens=[-1], return_scores=True,
return_no_speech_prob=True)
texts = []
for lang, r in zip(langs, results):
tokens = r.sequences_ids[0]
avg_logprob = r.scores[0] * len(tokens) / (len(tokens) + 1)
if r.no_speech_prob >= NO_SPEECH_THRESHOLD or avg_logprob + 1 < MIN_CONFIDENCE:
texts.append("")
else:
texts.append(self._tokenizer(lang).decode(tokens).strip())
return texts
plain = self._decode(enc, langs, opts.beam, None)
if not opts.hotwords:
return [text if keep_transcript(text, lp) else "" for text, lp in plain]
hinted = self._decode(enc, langs, opts.beam, " ".join(opts.hotwords))
return [choose_with_hint(ht, hlp, pt, plp, opts.hotwords) for (ht, hlp), (pt, plp) in zip(hinted, plain)]
def _decode(self, enc, langs: list[str], beam: int, hotwords: str | None) -> list[tuple[str, float]]:
m = self.model
prompts = [m.get_prompt(self._tokenizer(lang), [], without_timestamps=True, hotwords=hotwords)
for lang in langs]
results = m.model.generate(enc, prompts, beam_size=beam, max_length=224, suppress_blank=True,
suppress_tokens=[-1], return_scores=True)
# ``scores`` is the sequence log-probability divided by its length: the mean per token.
return [(self._tokenizer(lang).decode(r.sequences_ids[0]).strip(), float(r.scores[0]))
for lang, r in zip(langs, results)]

View File

@@ -74,7 +74,7 @@ STT settings (all optional in `hello` and `config`):
| `end_silence_ms` | 300 | Trailing VAD silence that ends an utterance. Range 100-2000. |
| `final_beam` | 2 | Beam size for finals (partials always use 1). Range 1-5. |
| `language` | `"ko"` | Whisper language code, or `null` to detect per utterance. |
| `hotwords` | `[]` | Words (wake words, names) passed to Whisper as a hint. |
| `hotwords` | `[]` | Words (wake words, names) passed to Whisper as a hint. The hint alone made Whisper write the wake word on laughter, so with hotwords every clip is also decoded without them and that plain decode decides (see Hallucination gate). |
### Utterance rules
@@ -82,10 +82,12 @@ Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filt
its own streaming Silero VAD (512-sample windows, state kept between frames).
- **Start**: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
- **speech_start** is sent once the utterance holds 250 ms of speech windows. An utterance that ends
before that is dropped without any message (a click or a cough).
- **speech_start** is sent once the utterance holds 250 ms of clearly voiced windows (probability at
or above 0.5). An utterance that ends before that is dropped without any message and never reaches
the model: a click, a cough, or noise.
- **Speech continues** while windows stay at or above 0.35 (hysteresis); a quieter window counts as
silence.
silence. Windows between 0.35 and 0.5 keep an utterance open but are not evidence of a voice, so
noise the VAD half-believes (fans, typing) never adds up to an utterance.
- **End**, with `end_reason`:
- `silence`: trailing silence reached `end_silence_ms`.
- `speaking_end`: the app sent `speaking_end`. Discord stops sending packets when a user stops
@@ -94,7 +96,12 @@ its own streaming Silero VAD (512-sample windows, state kept between frames).
- `max_length`: the utterance reached 28 seconds (Whisper's window). The next audio starts a new
utterance.
- The final clip keeps at most 200 ms of trailing silence.
- Before transcription the clip is peak-normalised to 0.9 with the gain capped at 10x.
- **Loudness equalisation** before transcription (partials and finals), so a quiet and a loud speaker
reach the model alike: rumble below 60 Hz is removed, the level is measured as the RMS of the voiced
windows only (silence, a click or a knock do not set the gain), the clip is brought to -20 dBFS with
the gain kept between -20 and +40 dB, and peaks above 0.8 are soft-limited below 0.98 instead of
clipping. Measured: speech from -60 dBFS to clipped shouting was transcribed equally well, and quiet
speech under loud clicks or knocks better than with peak normalisation.
### Scheduling
@@ -112,9 +119,35 @@ wait seconds behind it; two clips keep that wait to about one clip's time. With
captions fall behind and only the newest audio per speaker is transcribed.
Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has
already ended, or that is older than the last partial sent, is dropped. Whisper output is rejected
(empty text) when `no_speech_prob >= 0.5` or the confidence (`avg_logprob + 1`) is below 0.3, the
same gate as v1.
already ended, or that is older than the last partial sent, is dropped.
### Hallucination gate
Whisper writes text for voice-like sounds without words (coughs, laughter, a crying baby) and,
prompted with a hint, the hint itself. A transcript is kept only when all of these hold, else the
text is empty:
- It has letters or digits (not only punctuation).
- Its mean log-probability per token is at least -0.7. This is the decoder's own per-token mean; the
`avg_logprob` used by faster-whisper divides by one token more, which lifted a three-token
hallucination from -0.73 to -0.55 and let it through.
- Its compression ratio (UTF-8 bytes over zlib bytes) is at most 2.4, which catches repetition loops.
- With hotwords: the clip is decoded once with them and once without, from the same encoder pass.
The plain decode must pass the checks above, otherwise nothing was said. The hinted text is used
only when it stays within half the plain text's letters of it and mentions every hotword as often
as the plain one (a near miss in spelling counts as a mention), so the hint can fix how a spoken
wake word is written but never add or drop one. Otherwise the plain text is used.
`no_speech_prob` is not used: large-v3-turbo reports 0.0 for silence and noise alike.
Measured (2026-11-11, RTX 5050) with 54 real noise recordings from ESC-50 (typing, mouse, breathing,
coughing, laughing, crying, vacuum, ...), each at its own level and at -25 dBFS: of the 108 noise-only
clips, 102 never reached the model (the VAD rule above) and the 6 that did (coughs, laughter, crying)
all came out empty, with and without the wake word hint. Over 89 mixes of speech with that noise
(SNR 20, 10 and 0 dB, plus clean) the mean character error rate was 4.8%. Left over: one
"thank you" written for laughter just before a sentence (mean log-probability -0.70, right at the
line), two short words lost in loud noise, and one word repeated three times at SNR 0. A stricter
line (-0.6) would lose more real words than it saves.
### TTS

View File

@@ -5,7 +5,7 @@ from __future__ import annotations
import numpy as np
import pytest
from voice_worker.audio import InputConverter, peak_normalise
from voice_worker.audio import InputConverter, loudness_equalise
from voice_worker.protocol import (KIND_AUDIO, KIND_TTS, ProtocolError, decode_audio_frame, encode_audio_frame,
encode_tts_frame, decode_tts_frame)
@@ -104,19 +104,70 @@ def test_odd_byte_counts_are_carried_to_the_next_frame():
assert len(a) + len(b) == 10
def _db(x: np.ndarray) -> float:
return 20 * np.log10(np.sqrt(np.mean(np.asarray(x, dtype=np.float64) ** 2)) + 1e-12)
def _voice(seconds: float, db: float) -> np.ndarray:
"""A stand-in for a voice: a 220 Hz tone with harmonics, at ``db`` dBFS RMS."""
t = np.arange(int(16000 * seconds)) / 16000
x = sum(np.sin(2 * np.pi * 220 * k * t) / k for k in (1, 2, 3, 4))
return (x * 10 ** (db / 20) / np.sqrt(np.mean(x ** 2))).astype(np.float32)
def _mask(n_samples: int, voiced: slice) -> np.ndarray:
m = np.zeros(-(-n_samples // 512), dtype=bool)
m[voiced] = True
return m
@pytest.mark.unit
def test_quiet_clip_is_raised_to_the_target_peak():
x = np.full(100, 0.2, dtype=np.float32)
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)
@pytest.mark.parametrize("db", [-60, -45, -30, -10, -3])
def test_every_speaker_level_comes_out_at_the_same_loudness(db):
x = _voice(2.0, db)
y = loudness_equalise(x)
assert y.dtype == np.float32 and len(y) == len(x)
assert abs(_db(y) - (-20)) < 1.0
@pytest.mark.unit
def test_loudness_is_measured_on_the_voice_not_the_silence_around_it():
x = np.concatenate([np.zeros(16000, dtype=np.float32), _voice(1.0, -40)])
y = loudness_equalise(x, _mask(len(x), slice(32, None)))
assert abs(_db(y[16000:]) - (-20)) < 1.0
@pytest.mark.unit
def test_a_loud_click_neither_keeps_a_quiet_voice_quiet_nor_clips():
x = _voice(2.0, -50)
burst = np.random.default_rng(3).normal(0, 1, 200) * np.exp(-np.arange(200) / 30)
x[8000:8200] += (0.95 * burst / np.max(np.abs(burst))).astype(np.float32) # a mouse click, far louder
voiced = _mask(len(x), slice(None))
voiced[8000 // 512] = False
y = loudness_equalise(x, voiced)
assert abs(_db(y[:7000]) - (-20)) < 1.0
assert np.max(np.abs(y)) < 1.0
@pytest.mark.unit
def test_clipped_shouting_is_brought_down_without_new_clipping():
x = np.clip(_voice(2.0, -3) * 4, -1, 1)
y = loudness_equalise(x)
assert abs(_db(y) - (-20)) < 1.5
assert np.max(np.abs(y)) < 1.0
@pytest.mark.unit
def test_near_silence_is_not_boosted_past_the_gain_cap():
x = np.full(100, 0.001, dtype=np.float32)
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.01)
x = _voice(1.0, -90)
assert abs(_db(loudness_equalise(x)) - (-90 + 40)) < 1.0
@pytest.mark.unit
def test_loud_clip_is_brought_down_to_the_target_peak():
x = np.array([0.0, 1.0, -0.5], dtype=np.float32)
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)
def test_low_rumble_is_removed_before_levelling():
t = np.arange(32000) / 16000
rumble = (0.3 * np.sin(2 * np.pi * 30 * t)).astype(np.float32)
y = loudness_equalise(_voice(2.0, -40) + rumble)
spec = np.abs(np.fft.rfft(y))
freqs = np.fft.rfftfreq(len(y), 1 / 16000)
assert spec[np.argmin(np.abs(freqs - 30))] < 0.01 * spec[np.argmin(np.abs(freqs - 220))]

View File

@@ -0,0 +1,80 @@
"""Voice worker: which Whisper transcripts are kept (the hallucination gate), model-free."""
from __future__ import annotations
import pytest
from voice_worker.stt_whisper import MIN_MEAN_LOGPROB, choose_with_hint, compression_ratio, keep_transcript
@pytest.mark.unit
def test_a_confident_transcript_is_kept():
assert keep_transcript("자비스, 오늘 서울 날씨 어때?", -0.05)
@pytest.mark.unit
def test_low_confidence_is_rejected_even_for_a_short_output():
# A three-token hallucination on laughter scored a mean of -0.73 per token; the length
# correction used before (len / (len + 1)) lifted it to -0.55 and let it through.
assert not keep_transcript("자비스", MIN_MEAN_LOGPROB - 0.03)
assert keep_transcript("자비스", MIN_MEAN_LOGPROB + 0.03)
@pytest.mark.unit
def test_empty_and_punctuation_only_outputs_are_rejected():
assert not keep_transcript("", -0.01)
assert not keep_transcript(" . . ", -0.01)
@pytest.mark.unit
def test_a_repetition_loop_is_rejected():
loop = "자막 제공 및 " * 12
assert compression_ratio(loop) > 2.4
assert not keep_transcript(loop, -0.1)
assert compression_ratio("오늘 회의는 오후 세 시에 시작해서 다섯 시쯤 끝날 것 같아요.") < 2.4
HINT = ("자비스",)
ASK = "지금 두 번째 모니터에 뭐 떠 있는지 알려줄래?"
@pytest.mark.unit
def test_voice_like_noise_that_the_hint_turns_into_the_wake_word_gives_nothing():
# Laughter, decoded with the wake word as a hint and without it.
assert choose_with_hint("자비스", -0.4, "감사합니다.", -1.4, HINT) == ""
assert choose_with_hint("자비스", -0.4, "", -0.1, HINT) == ""
@pytest.mark.unit
def test_a_wake_word_the_hint_adds_to_real_speech_is_dropped():
assert choose_with_hint(ASK + " 자비스", -0.1, ASK, -0.1, HINT) == ASK
@pytest.mark.unit
def test_the_hint_may_fix_how_a_spoken_wake_word_is_spelt():
fixed = "자비스, 오늘 서울 날씨 어때?"
assert choose_with_hint(fixed, -0.1, "자비수, 오늘 서울 날씨 어때?", -0.2, HINT) == fixed
@pytest.mark.unit
def test_a_wake_word_the_hint_loses_is_kept_from_the_plain_decode():
plain = "자비스, 오늘 서울 날씨 어때?"
assert choose_with_hint("오늘 서울 날씨 어때?", -0.2, plain, -0.07, HINT) == plain
@pytest.mark.unit
def test_an_unsure_hint_decode_falls_back_to_the_plain_one():
assert choose_with_hint(ASK, MIN_MEAN_LOGPROB - 0.2, ASK, -0.1, HINT) == ASK
assert choose_with_hint("zzz", -0.1, ASK, -0.1, HINT) == ASK
@pytest.mark.unit
def test_wake_word_matching_ignores_case_spacing_and_punctuation():
assert choose_with_hint("Hey Jarvis, lights off.", -0.1, "hey jarvis lights off", -0.1, ("jarvis",)) \
== "Hey Jarvis, lights off."
@pytest.mark.unit
def test_spacing_and_punctuation_differences_do_not_change_the_choice():
assert choose_with_hint("자비스 오늘 서울 날씨 어때", -0.1, "자비스, 오늘 서울 날씨 어때?", -0.1, HINT) \
== "자비스 오늘 서울 날씨 어때"

View File

@@ -171,7 +171,10 @@ def test_tts_reply_comes_back_as_the_same_words(engines):
if engines[2] is None:
pytest.skip("set JARVIS_MELO_PYTHON to the MeloTTS venv python")
from voice_worker.protocol import decode_tts_frame, encode_audio_frame
text = "오늘 서울은 맑고 기온은 이십삼 도예요. 우산은 필요 없어요."
# MeloTTS samples its voice differently each time and blurs some words ("기온은 이십삼" came back
# as "비오는 이시찬" in raw, peak-normalised and equalised audio alike), so the sentence is one it
# says clearly; this checks the worker's plumbing, not MeloTTS's diction.
text = "오늘 서울은 맑고 따뜻해요. 우산은 집에 두고 가셔도 돼요."
async def body(ws):
await ws.send(json.dumps({"type": "config", "partials": False}))
@@ -207,3 +210,106 @@ def test_tts_reply_comes_back_as_the_same_words(engines):
print(f"tts chunks={len(chunks)} first_chunk_s={first:.2f} heard={heard}")
assert [c[1] for c in chunks] == [0, 1]
assert cer(text, heard) <= 0.2
def _stream_clip_16k(engines, clip: np.ndarray, stt: dict) -> list[dict]:
"""Stream one 16 kHz mono clip as a single speaker (as fast as the socket takes it), end it,
and return every final the worker sends."""
from voice_worker.protocol import encode_audio_frame
pcm = (np.clip(to_48k_stereo(clip), -1, 1) * 32767).astype("<i2").tobytes()
step = 48000 * FRAME_MS // 1000 * 2 * 2
async def body(ws):
await ws.send(json.dumps({"type": "config", **stt}))
for k in range(0, len(pcm), step):
await ws.send(encode_audio_frame("s", pcm[k:k + step]))
await ws.send(json.dumps({"type": "speaking_end", "speaker": "s"}))
finals = []
while True:
try:
msg = json.loads(await asyncio.wait_for(ws.recv(), 3))
except asyncio.TimeoutError:
return finals
if msg["type"] == "final":
finals.append(msg)
return asyncio.run(with_worker(engines, body))
def _babble(seconds: float, rng) -> np.ndarray:
"""Speech-like noise: the kit's Korean clips played backwards and overlaid (voice-like, no words)."""
n = int(16000 * seconds)
x = np.zeros(n, dtype=np.float32)
for name in ("ko_01.wav", "ko_03.wav", "ko_05.wav", "ko_07.wav"):
c = load(name)[::-1]
x += np.roll(np.tile(c, n // len(c) + 1)[:n], int(rng.integers(0, n)))
return x
def _at_db(x: np.ndarray, db: float) -> np.ndarray:
return (x * 10 ** (db / 20) / (np.sqrt(np.mean(x.astype(np.float64) ** 2)) + 1e-12)).astype(np.float32)
@pytest.mark.parametrize("level_db", [-40, -25])
def test_voice_like_noise_with_the_wake_word_hint_produces_no_text(engines, level_db):
"""Backwards babble passes the VAD. Given the wake word as a hint, Whisper used to write the wake
word (a phantom call to the bot); without words in the audio no final may carry text."""
rng = np.random.default_rng(level_db + 100)
finals = _stream_clip_16k(engines, _at_db(_babble(6.0, rng), level_db),
{"hotwords": ["자비스"], "partials": False})
print(level_db, [(m["text"], m["end_reason"]) for m in finals])
assert all(m["text"] == "" for m in finals)
@pytest.mark.parametrize("snr_db", [20, 10])
def test_speech_over_voice_like_noise_keeps_the_words(engines, snr_db):
rng = np.random.default_rng(snr_db)
refs = json.loads((SAMPLES / "refs.json").read_text(encoding="utf-8"))["samples"]
errs = []
for r in refs[:4]:
speech = np.concatenate([np.zeros(8000, np.float32), _at_db(load(r["file"]), -25), np.zeros(8000, np.float32)])
noisy = speech + _at_db(_babble(len(speech) / 16000, rng), -25 - snr_db)
finals = _stream_clip_16k(engines, noisy, {"hotwords": ["자비스"], "partials": False})
text = " ".join(m["text"] for m in finals).strip()
errs.append(cer(r["text"], text))
print(snr_db, r["file"], round(errs[-1], 2), text)
assert np.mean(errs) <= 0.15
@pytest.mark.parametrize("level_db", [-60, -45, -10])
def test_quiet_and_loud_speakers_are_both_understood(engines, level_db):
refs = json.loads((SAMPLES / "refs.json").read_text(encoding="utf-8"))["samples"]
errs = []
for r in refs[:4]:
x = np.concatenate([np.zeros(8000, np.float32), _at_db(load(r["file"]), level_db), np.zeros(8000, np.float32)])
if level_db == -10:
x = np.clip(x * 4, -1, 1) # shouting into a hot mic: clipped
finals = _stream_clip_16k(engines, x, {"partials": False})
errs.append(cer(r["text"], " ".join(m["text"] for m in finals)))
print(level_db, [round(e, 2) for e in errs])
assert np.mean(errs) <= 0.15
def test_real_noise_recordings_produce_no_text(engines):
"""Optional: point JARVIS_NOISE_DIR at noise-only recordings (16 kHz or 44.1 kHz mono/stereo WAV,
e.g. ESC-50 keyboard, mouse, breathing, coughing, laughing). None may produce text."""
folder = os.environ.get("JARVIS_NOISE_DIR")
if not folder:
pytest.skip("set JARVIS_NOISE_DIR to a folder of noise-only WAV files")
from scipy.signal import resample_poly
texts = []
for path in sorted(Path(folder).glob("*.wav")):
with wave.open(str(path), "rb") as w:
x = np.frombuffer(w.readframes(w.getnframes()), dtype="<i2").astype(np.float32) / 32768.0
if w.getnchannels() == 2:
x = x.reshape(-1, 2).mean(axis=1)
rate = w.getframerate()
if rate != 16000:
from math import gcd
g = gcd(rate, 16000)
x = resample_poly(x, 16000 // g, rate // g).astype(np.float32)
for m in _stream_clip_16k(engines, x, {"hotwords": ["자비스"], "partials": False}):
if m["text"]:
texts.append((path.name, m["text"]))
print(texts)
assert texts == []

View File

@@ -102,6 +102,30 @@ def test_hysteresis_keeps_speech_going_but_never_starts_it():
assert len(f.of(Final)) == 1
@pytest.mark.unit
def test_weak_speech_like_noise_never_builds_up_to_an_utterance():
"""A breath of real speech probability followed by a long stretch of 'maybe speech' (fans,
keyboard, background rumble that the VAD scores around 0.4) is not enough to call the model."""
f = Feeder()
for _ in range(3):
f.feed(0.9, 40)
f.feed(0.4, 1000)
f.feed(0.0, 500)
assert f.events == []
@pytest.mark.unit
def test_finals_mark_which_windows_were_voiced():
f = Feeder()
f.feed(0.0, 500)
f.feed(0.9, 640)
f.feed(0.0, 500)
fin = f.of(Final)[0]
assert len(fin.voiced) == -(-len(fin.audio) // 512)
assert 20 <= fin.voiced.sum() <= 21 # 640 ms of 32 ms windows, not aligned to the frames
assert not fin.voiced[0] and not fin.voiced[-1] # pre-roll and tail are not voiced
@pytest.mark.unit
def test_speaking_end_closes_the_utterance_at_once():
f = Feeder()

View File

@@ -1,7 +1,8 @@
"""Voice worker end to end over a real local WebSocket, with fake STT/VAD/TTS engines.
The fake VAD reads speech probability off the amplitude and the fake transcriber names a clip by its
peak level, so each speaker's text tells which speaker's audio reached the model.
"level": each speaker sends a tone whose amplitude is the level (what the fake VAD reads) and whose
pitch is level x 1000 Hz (what the fake transcriber reads, since the worker evens out loudness).
"""
from __future__ import annotations
@@ -33,7 +34,7 @@ class LevelTranscriber:
def transcribe(self, clips, opts):
self.calls.append((opts, len(clips)))
return [f"level {np.median(np.abs(c)):.1f}" for c in clips]
return [f"level {np.argmax(np.abs(np.fft.rfft(c))) * 16000 / len(c) / 1000:.1f}" for c in clips]
class FakeTts:
@@ -54,7 +55,9 @@ class FakeTts:
def pcm(level: float, ms: int, rate=16000, channels=1) -> bytes:
n = rate * ms // 1000
return (np.full(n * channels, level) * 32767).astype("<i2").tobytes()
# 20 ms frames hold whole cycles at these pitches, so frames join without a click.
x = level * np.sin(2 * np.pi * level * 1000 * np.arange(n) / rate)
return (np.repeat(x, channels) * 32767).astype("<i2").tobytes()
async def with_worker(body, transcriber=None, tts=None, **kw):