fix(voice_worker): even out speaker loudness and stop transcribing noise into words
- Loudness equalisation replaces peak normalisation: rumble below 60 Hz removed, level measured on the VAD-voiced windows only, brought to -20 dBFS (gain -20..+40 dB), soft limiter below 0.98. - Only clearly voiced windows (VAD >= 0.5) count towards the 250 ms needed before an utterance reaches the model, so noise the VAD half-believes never calls Whisper. - Hallucination gate: mean token log-probability >= -0.7 (the old len/(len+1) correction let short hallucinations through), compression ratio <= 2.4; no_speech_prob dropped (always 0.0 on turbo). - With hotwords, the plain decode of the same encoder pass decides; the hint may fix a wake word's spelling but cannot add or drop one (it wrote the wake word on laughter and after real sentences). - Opt-in real-model evals: voice-like babble with the hint, speech over babble, quiet and clipped speakers, and a JARVIS_NOISE_DIR folder of real noise recordings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -22,7 +22,9 @@
|
||||
- **2단 디코딩**: 부분 전사는 beam 1(빠름), 최종 전사는 beam 2~3(정확도)입니다.
|
||||
- **게임 중 LLM이 답하는 동안**: STT와 LLM은 같은 GPU를 씁니다. 10명이 쉬지 않고 말하며 부분 전사를 계속 돌리면 LLM 출력이 1/2.5~1/3로 줄었습니다(M0 `gamemode_stt`). 그래서 게임 중에 LLM이 답을 만드는 동안에는 부분 전사를 멈추고(그동안 실시간 자막이 바뀌지 않음), 말이 끝난 뒤의 최종 전사는 기다리지 않고 바로 처리합니다. 게임이 없을 때는 LLM이 답하는 동안에도 부분 전사를 그대로 합니다(사용자 결정, 2026-11-10). 최종 전사가 부분 전사보다 먼저입니다. 멈추기로 한 것은 M0 `gamemode_tune` 측정 결과입니다(2026-11-11): 2초 간격은 37.5 tok/s, 음성 첫 문장 0.89초에 최종 전사 대기가 최대 1.2초였고, 멈추면 64.9 tok/s, 0.55초였습니다. 부분 전사를 멈췄을 때 beam 2~3 최종 전사와 긴 말의 실제 지연은 아직 재지 않았습니다([pc-features.md](pc-features.md#자동-조절-측정-m0-gamemode_tune-2026-11-11)).
|
||||
- **부분 전사는 2명씩만 묶음(M1 실측)**: Whisper 인코더는 말 길이와 상관없이 늘 30초 창을 계산해서, 묶어도 비용이 사람 수에 비례합니다(RTX 5050에서 1명 0.2초, 10명 2.1초). 10명분 부분 전사를 한 번에 돌리면 그동안 들어온 최종 전사가 2초 넘게 기다려서, 부분 전사는 한 번에 2명까지만 돌리고 그 사이에 최종 전사가 먼저 들어가게 했습니다. 여러 명이 함께 말하면 자막은 늦어지고 사람마다 가장 최근 소리만 전사합니다. 인코더 창을 줄이면 4~7배 빨라졌지만 같은 문장을 되풀이하는 오류가 생겨 쓰지 않습니다.
|
||||
- **호출어 힌트**: 등록한 호출어를 STT에 힌트로 넘겨 잘못 알아듣는 일을 줄입니다.
|
||||
- **호출어 힌트**: 등록한 호출어를 STT에 힌트로 넘겨 잘못 알아듣는 일을 줄입니다. 힌트만 쓰면 웃음소리에서 "자비스"를 지어내거나 실제 문장 끝에 "자비스"를 붙여서(M1 측정, 2026-11-11), 힌트가 있으면 같은 소리를 힌트 없이도 한 번 더 해석해 그 결과로 판단합니다. 힌트는 호출어 철자를 고치는 데만 쓰이고 호출어를 더하거나 빼지 못합니다.
|
||||
- **음량 맞추기**: 사람마다 목소리 크기가 달라도 같은 크기로 맞춰 STT에 넣습니다. 목소리가 있는 구간만 재서 -20dBFS로 맞추고(최대 +40dB, 낮은 웅웅거림 제거, 큰 소리는 찌그러지지 않게 부드럽게 눌러 줌), 클릭이나 노크 소리가 음량을 정하지 않게 합니다. 측정하면 -60dBFS 속삭임부터 찌그러질 만큼 큰 소리까지 똑같이 받아 적었습니다.
|
||||
- **잡음 거르기와 헛 전사 막기**: 목소리가 분명한 구간(음성 확률 0.5 이상)이 0.25초 넘게 있어야 STT를 돌립니다. 키보드, 마우스, 발소리, 청소기 같은 잡음은 여기서 걸러집니다. 기침, 웃음처럼 목소리 같은 소리는 STT까지 가지만 결과의 토큰당 평균 확신도(-0.7 이상)와 반복 여부로 거릅니다. 실제 잡음 54개(ESC-50)를 원래 크기와 -25dBFS로 넣은 108개 중 102개는 STT까지 가지 않았고, STT까지 간 6개(기침, 웃음, 울음)도 모두 빈 결과였습니다(호출어 힌트가 있을 때도 같음). 그 잡음 위에 말한 문장 89개는 문자 오류율 평균 4.8%였고, 남은 문제는 웃음 직후 "감사합니다." 한 번(확신도가 기준선에 걸침), 시끄러운 잡음 속 짧은 단어 2개 누락, 같은 단어 세 번 반복 한 번입니다. 이 모델(large-v3-turbo)의 "말 없음" 확률은 늘 0이라 쓰지 않습니다.
|
||||
- **RealtimeSTT 사용 범위**: RealtimeSTT는 화자 하나에 인스턴스 하나 구조라 10명이면 모델을 10번 올립니다. 그래서 설계(부분/최종 전사, VAD 결합)만 참고하고, **모델 1개를 10명이 공유하는 배치 방식**으로 직접 구현합니다.
|
||||
- **선택지**: 더 낮은 지연이 필요하면 sherpa-onnx 스트리밍 모델을 "초고속" 옵션으로 검토합니다(한국어 정확도 확인 후).
|
||||
|
||||
|
||||
@@ -50,9 +50,39 @@ class InputConverter:
|
||||
return out.astype(np.float32)
|
||||
|
||||
|
||||
def peak_normalise(x: np.ndarray, target: float = 0.9, max_gain: float = 10.0) -> np.ndarray:
|
||||
peak = float(np.max(np.abs(x))) if len(x) else 0.0
|
||||
if peak <= 0:
|
||||
return x.astype(np.float32)
|
||||
gain = min(target / peak, max_gain)
|
||||
return (x * gain).astype(np.float32)
|
||||
TARGET_DB = -20.0 # loudness every speaker is brought to, dBFS RMS over the voiced windows
|
||||
MAX_GAIN_DB = 40.0 # a whisper at -60 dBFS still reaches the target; pure hiss is not blown up
|
||||
MIN_GAIN_DB = -20.0
|
||||
HIGHPASS_HZ = 60.0 # mains hum and desk rumble carry no speech
|
||||
_KNEE = 0.8 # soft limiter: peaks above this bend smoothly towards the ceiling instead of clipping
|
||||
_CEILING = 0.98
|
||||
_WINDOW = 512
|
||||
|
||||
|
||||
def loudness_equalise(x: np.ndarray, voiced: np.ndarray | None = None) -> np.ndarray:
|
||||
"""Bring a clip to the same loudness whoever spoke it, without clipping.
|
||||
|
||||
``voiced`` marks the 512-sample windows the VAD heard as speech; the level is measured on them so
|
||||
silence, a click or a knock does not decide the gain. Without it the louder half of the windows is
|
||||
used. Low rumble is removed first and peaks are soft-limited after the gain.
|
||||
"""
|
||||
x = np.asarray(x, dtype=np.float32)
|
||||
if len(x) == 0:
|
||||
return x
|
||||
spec = np.fft.rfft(x.astype(np.float64))
|
||||
spec[np.fft.rfftfreq(len(x), 1 / TARGET_RATE) < HIGHPASS_HZ] = 0
|
||||
y = np.fft.irfft(spec, len(x))
|
||||
n_win = -(-len(y) // _WINDOW)
|
||||
frames = np.pad(y, (0, n_win * _WINDOW - len(y))).reshape(n_win, _WINDOW)
|
||||
energy = (frames ** 2).mean(axis=1)
|
||||
if voiced is not None and len(voiced) == n_win and voiced.any():
|
||||
sel = energy[voiced]
|
||||
else:
|
||||
sel = energy[energy >= np.median(energy)]
|
||||
level_db = 10 * np.log10(sel.mean() + 1e-20)
|
||||
gain_db = float(np.clip(TARGET_DB - level_db, MIN_GAIN_DB, MAX_GAIN_DB))
|
||||
y = y * 10 ** (gain_db / 20)
|
||||
over = np.abs(y) > _KNEE
|
||||
room = _CEILING - _KNEE
|
||||
y[over] = np.sign(y[over]) * (_KNEE + room * np.tanh((np.abs(y[over]) - _KNEE) / room))
|
||||
return y.astype(np.float32)
|
||||
|
||||
@@ -33,6 +33,7 @@ class SpeechStart:
|
||||
class Partial:
|
||||
utt: int
|
||||
audio: np.ndarray
|
||||
voiced: np.ndarray # one bool per 512-sample window: the VAD heard speech there
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -41,6 +42,7 @@ class Final:
|
||||
audio: np.ndarray
|
||||
reason: str # silence | speaking_end | idle | max_length
|
||||
decided_at: float
|
||||
voiced: np.ndarray
|
||||
|
||||
|
||||
def _windows(ms: float) -> int:
|
||||
@@ -60,7 +62,8 @@ class Segmenter:
|
||||
def _reset(self) -> None:
|
||||
self._active = False
|
||||
self._win: list[np.ndarray] = []
|
||||
self._speech = 0
|
||||
self._voiced: list[bool] = []
|
||||
self._speech = 0 # windows that clearly held speech (>= START_PROB)
|
||||
self._trailing = 0
|
||||
self._started = False
|
||||
self._last_partial_at = 0.0
|
||||
@@ -83,7 +86,7 @@ class Segmenter:
|
||||
and now - self._last_partial_at >= self.settings.partial_interval_ms / 1000:
|
||||
self._last_partial_at = now
|
||||
self._last_partial_len = len(self._win)
|
||||
events.append(Partial(self._utt_count, np.concatenate(self._win)))
|
||||
events.append(Partial(self._utt_count, np.concatenate(self._win), np.array(self._voiced)))
|
||||
return events
|
||||
|
||||
def _window(self, w: np.ndarray, now: float) -> list:
|
||||
@@ -92,6 +95,7 @@ class Segmenter:
|
||||
if p >= START_PROB:
|
||||
self._active = True
|
||||
self._win = self._pre + [w]
|
||||
self._voiced = [False] * len(self._pre) + [True]
|
||||
self._pre = []
|
||||
self._speech = 1
|
||||
self._trailing = 0
|
||||
@@ -100,8 +104,12 @@ class Segmenter:
|
||||
del self._pre[:-_windows(PRE_ROLL_MS)]
|
||||
return []
|
||||
self._win.append(w)
|
||||
if p >= KEEP_PROB:
|
||||
self._voiced.append(p >= START_PROB)
|
||||
# "Maybe speech" (between the two thresholds) keeps an utterance going but is not evidence
|
||||
# of a voice, so noise the VAD half-believes never adds up to a call to the model.
|
||||
if p >= START_PROB:
|
||||
self._speech += 1
|
||||
if p >= KEEP_PROB:
|
||||
self._trailing = 0
|
||||
else:
|
||||
self._trailing += 1
|
||||
@@ -121,8 +129,9 @@ class Segmenter:
|
||||
events: list = []
|
||||
if self._active and self._started:
|
||||
drop = max(0, self._trailing - _windows(TAIL_KEEP_MS))
|
||||
kept = self._win[:len(self._win) - drop]
|
||||
events.append(Final(self._utt_count, np.concatenate(kept), reason, now))
|
||||
keep = len(self._win) - drop
|
||||
events.append(Final(self._utt_count, np.concatenate(self._win[:keep]), reason, now,
|
||||
np.array(self._voiced[:keep])))
|
||||
self._reset()
|
||||
return events
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ import numpy as np
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
from .audio import InputConverter
|
||||
from .audio import InputConverter, loudness_equalise
|
||||
from .protocol import KIND_AUDIO, ProtocolError, decode_audio_frame, encode_tts_frame
|
||||
from .scheduler import DecodeOptions, SttResult, SttScheduler, Transcriber
|
||||
from .segmenter import SR, Final, Partial, Segmenter, SpeechStart, VadStream
|
||||
@@ -211,7 +211,7 @@ class Session:
|
||||
self.send({"type": "speech_start", "speaker": speaker, "utt": ev.utt})
|
||||
elif isinstance(ev, Partial):
|
||||
if sched and self.settings.partials:
|
||||
sched.submit_partial((self.sid, speaker), ev.audio, self._opts(1),
|
||||
sched.submit_partial((self.sid, speaker), loudness_equalise(ev.audio, ev.voiced), self._opts(1),
|
||||
self._partial_cb(speaker, st, ev.utt))
|
||||
elif isinstance(ev, Final):
|
||||
debug_log(f"utterance end speaker={speaker} utt={ev.utt} reason={ev.reason} "
|
||||
@@ -219,7 +219,8 @@ class Session:
|
||||
if sched is None:
|
||||
self._send_final(speaker, ev, SttResult("", 0, "stt unavailable"))
|
||||
else:
|
||||
sched.submit_final((self.sid, speaker), ev.audio, self._opts(self.settings.final_beam),
|
||||
sched.submit_final((self.sid, speaker), loudness_equalise(ev.audio, ev.voiced),
|
||||
self._opts(self.settings.final_beam),
|
||||
lambda res, ev=ev: self._send_final(speaker, ev, res))
|
||||
|
||||
def _partial_cb(self, speaker: str, st: _Speaker, utt: int):
|
||||
|
||||
@@ -4,18 +4,83 @@ from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import zlib
|
||||
|
||||
import numpy as np
|
||||
|
||||
from jarvis.debug import debug_log
|
||||
|
||||
from .audio import peak_normalise
|
||||
from .scheduler import DecodeOptions
|
||||
|
||||
SR = 16000
|
||||
NO_SPEECH_THRESHOLD = 0.5 # same gate as v1 (whisper_no_speech_threshold)
|
||||
MIN_CONFIDENCE = 0.3 # avg_logprob + 1, same as v1 (whisper_min_confidence)
|
||||
# Mean log-probability per output token. Measured on real noise (coughs, laughter, crying) and speech
|
||||
# over it: every hallucination stayed below, real speech above except two short words in loud noise.
|
||||
# Whisper's no_speech_prob is not used: large-v3-turbo reports 0.0 for silence and noise alike.
|
||||
MIN_MEAN_LOGPROB = -0.7
|
||||
MAX_COMPRESSION_RATIO = 2.4 # repetition loops ("A B A B A B ...") compress far better than speech
|
||||
|
||||
|
||||
def _letters(text: str) -> str:
|
||||
"""Text without spacing, punctuation and case, so decodes compare by what was said."""
|
||||
return re.sub(r"[\W_]+", "", text).casefold()
|
||||
|
||||
|
||||
def compression_ratio(text: str) -> float:
|
||||
raw = text.encode("utf-8")
|
||||
return len(raw) / len(zlib.compress(raw)) if raw else 0.0
|
||||
|
||||
|
||||
def keep_transcript(text: str, mean_logprob: float) -> bool:
|
||||
return bool(_letters(text)) and mean_logprob >= MIN_MEAN_LOGPROB \
|
||||
and compression_ratio(text) <= MAX_COMPRESSION_RATIO
|
||||
|
||||
|
||||
def _edit_distance(a: str, b: str) -> int:
|
||||
prev = list(range(len(b) + 1))
|
||||
for i, ca in enumerate(a, 1):
|
||||
cur = [i]
|
||||
for j, cb in enumerate(b, 1):
|
||||
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
|
||||
prev = cur
|
||||
return prev[-1]
|
||||
|
||||
|
||||
def _mentions(text: str, word: str) -> int:
|
||||
"""How many times ``word`` occurs in ``text``, allowing a near miss in spelling (자비수 for 자비스)."""
|
||||
t, w = _letters(text), _letters(word)
|
||||
if not w:
|
||||
return 0
|
||||
slack = max(1, len(w) // 3)
|
||||
count, i = 0, 0
|
||||
while i <= len(t) - max(1, len(w) - slack):
|
||||
hit = next((n for n in range(len(w) - slack, len(w) + slack + 1)
|
||||
if i + n <= len(t) and _edit_distance(t[i:i + n], w) <= slack), None)
|
||||
if hit:
|
||||
count += 1
|
||||
i += hit
|
||||
else:
|
||||
i += 1
|
||||
return count
|
||||
|
||||
|
||||
def choose_with_hint(hint_text: str, hint_lp: float, plain_text: str, plain_lp: float,
|
||||
hotwords: tuple[str, ...]) -> str:
|
||||
"""Pick between the decode that had the hotwords as a hint and the plain decode of the same audio.
|
||||
|
||||
The hint can make Whisper write a wake word that nobody said (on laughter, or tacked onto a real
|
||||
sentence), so the plain decode decides: no plain text means nothing was said, and the hint's text
|
||||
is used only when it stays close to the plain one and mentions every hotword as often (it may fix
|
||||
their spelling, not add or drop them).
|
||||
"""
|
||||
if not keep_transcript(plain_text, plain_lp):
|
||||
return ""
|
||||
close = _edit_distance(_letters(hint_text), _letters(plain_text)) <= len(_letters(plain_text)) // 2
|
||||
if close and keep_transcript(hint_text, hint_lp) and all(
|
||||
_mentions(hint_text, w) == _mentions(plain_text, w) for w in hotwords):
|
||||
return hint_text
|
||||
return plain_text
|
||||
|
||||
|
||||
def add_cuda_dll_dirs() -> None:
|
||||
@@ -50,11 +115,12 @@ class WhisperTranscriber:
|
||||
self.transcribe([np.zeros(SR, dtype=np.float32)], DecodeOptions(beam=1, language="en", hotwords=()))
|
||||
|
||||
def transcribe(self, clips: list[np.ndarray], opts: DecodeOptions) -> list[str]:
|
||||
"""Clips arrive loudness-equalised by the session."""
|
||||
m = self.model
|
||||
n_frames = m.feature_extractor.nb_max_frames
|
||||
feats = []
|
||||
for c in clips:
|
||||
a = peak_normalise(c[: SR * 30])
|
||||
a = np.asarray(c[: SR * 30], dtype=np.float32)
|
||||
a = np.pad(a, (0, SR * 30 - len(a)))
|
||||
feats.append(m.feature_extractor(a, padding=0)[:, :n_frames])
|
||||
enc = m.encode(np.stack(feats).astype(np.float32))
|
||||
@@ -62,17 +128,18 @@ class WhisperTranscriber:
|
||||
langs = [opts.language] * len(clips)
|
||||
else:
|
||||
langs = [r[0][0][2:-2] for r in m.model.detect_language(enc)]
|
||||
hot = " ".join(opts.hotwords) or None
|
||||
prompts = [m.get_prompt(self._tokenizer(lang), [], without_timestamps=True, hotwords=hot) for lang in langs]
|
||||
results = m.model.generate(enc, prompts, beam_size=opts.beam, max_length=224,
|
||||
suppress_blank=True, suppress_tokens=[-1], return_scores=True,
|
||||
return_no_speech_prob=True)
|
||||
texts = []
|
||||
for lang, r in zip(langs, results):
|
||||
tokens = r.sequences_ids[0]
|
||||
avg_logprob = r.scores[0] * len(tokens) / (len(tokens) + 1)
|
||||
if r.no_speech_prob >= NO_SPEECH_THRESHOLD or avg_logprob + 1 < MIN_CONFIDENCE:
|
||||
texts.append("")
|
||||
else:
|
||||
texts.append(self._tokenizer(lang).decode(tokens).strip())
|
||||
return texts
|
||||
plain = self._decode(enc, langs, opts.beam, None)
|
||||
if not opts.hotwords:
|
||||
return [text if keep_transcript(text, lp) else "" for text, lp in plain]
|
||||
hinted = self._decode(enc, langs, opts.beam, " ".join(opts.hotwords))
|
||||
return [choose_with_hint(ht, hlp, pt, plp, opts.hotwords) for (ht, hlp), (pt, plp) in zip(hinted, plain)]
|
||||
|
||||
def _decode(self, enc, langs: list[str], beam: int, hotwords: str | None) -> list[tuple[str, float]]:
|
||||
m = self.model
|
||||
prompts = [m.get_prompt(self._tokenizer(lang), [], without_timestamps=True, hotwords=hotwords)
|
||||
for lang in langs]
|
||||
results = m.model.generate(enc, prompts, beam_size=beam, max_length=224, suppress_blank=True,
|
||||
suppress_tokens=[-1], return_scores=True)
|
||||
# ``scores`` is the sequence log-probability divided by its length: the mean per token.
|
||||
return [(self._tokenizer(lang).decode(r.sequences_ids[0]).strip(), float(r.scores[0]))
|
||||
for lang, r in zip(langs, results)]
|
||||
|
||||
@@ -74,7 +74,7 @@ STT settings (all optional in `hello` and `config`):
|
||||
| `end_silence_ms` | 300 | Trailing VAD silence that ends an utterance. Range 100-2000. |
|
||||
| `final_beam` | 2 | Beam size for finals (partials always use 1). Range 1-5. |
|
||||
| `language` | `"ko"` | Whisper language code, or `null` to detect per utterance. |
|
||||
| `hotwords` | `[]` | Words (wake words, names) passed to Whisper as a hint. |
|
||||
| `hotwords` | `[]` | Words (wake words, names) passed to Whisper as a hint. The hint alone made Whisper write the wake word on laughter, so with hotwords every clip is also decoded without them and that plain decode decides (see Hallucination gate). |
|
||||
|
||||
### Utterance rules
|
||||
|
||||
@@ -82,10 +82,12 @@ Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filt
|
||||
its own streaming Silero VAD (512-sample windows, state kept between frames).
|
||||
|
||||
- **Start**: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
|
||||
- **speech_start** is sent once the utterance holds 250 ms of speech windows. An utterance that ends
|
||||
before that is dropped without any message (a click or a cough).
|
||||
- **speech_start** is sent once the utterance holds 250 ms of clearly voiced windows (probability at
|
||||
or above 0.5). An utterance that ends before that is dropped without any message and never reaches
|
||||
the model: a click, a cough, or noise.
|
||||
- **Speech continues** while windows stay at or above 0.35 (hysteresis); a quieter window counts as
|
||||
silence.
|
||||
silence. Windows between 0.35 and 0.5 keep an utterance open but are not evidence of a voice, so
|
||||
noise the VAD half-believes (fans, typing) never adds up to an utterance.
|
||||
- **End**, with `end_reason`:
|
||||
- `silence`: trailing silence reached `end_silence_ms`.
|
||||
- `speaking_end`: the app sent `speaking_end`. Discord stops sending packets when a user stops
|
||||
@@ -94,7 +96,12 @@ its own streaming Silero VAD (512-sample windows, state kept between frames).
|
||||
- `max_length`: the utterance reached 28 seconds (Whisper's window). The next audio starts a new
|
||||
utterance.
|
||||
- The final clip keeps at most 200 ms of trailing silence.
|
||||
- Before transcription the clip is peak-normalised to 0.9 with the gain capped at 10x.
|
||||
- **Loudness equalisation** before transcription (partials and finals), so a quiet and a loud speaker
|
||||
reach the model alike: rumble below 60 Hz is removed, the level is measured as the RMS of the voiced
|
||||
windows only (silence, a click or a knock do not set the gain), the clip is brought to -20 dBFS with
|
||||
the gain kept between -20 and +40 dB, and peaks above 0.8 are soft-limited below 0.98 instead of
|
||||
clipping. Measured: speech from -60 dBFS to clipped shouting was transcribed equally well, and quiet
|
||||
speech under loud clicks or knocks better than with peak normalisation.
|
||||
|
||||
### Scheduling
|
||||
|
||||
@@ -112,9 +119,35 @@ wait seconds behind it; two clips keep that wait to about one clip's time. With
|
||||
captions fall behind and only the newest audio per speaker is transcribed.
|
||||
|
||||
Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has
|
||||
already ended, or that is older than the last partial sent, is dropped. Whisper output is rejected
|
||||
(empty text) when `no_speech_prob >= 0.5` or the confidence (`avg_logprob + 1`) is below 0.3, the
|
||||
same gate as v1.
|
||||
already ended, or that is older than the last partial sent, is dropped.
|
||||
|
||||
### Hallucination gate
|
||||
|
||||
Whisper writes text for voice-like sounds without words (coughs, laughter, a crying baby) and,
|
||||
prompted with a hint, the hint itself. A transcript is kept only when all of these hold, else the
|
||||
text is empty:
|
||||
|
||||
- It has letters or digits (not only punctuation).
|
||||
- Its mean log-probability per token is at least -0.7. This is the decoder's own per-token mean; the
|
||||
`avg_logprob` used by faster-whisper divides by one token more, which lifted a three-token
|
||||
hallucination from -0.73 to -0.55 and let it through.
|
||||
- Its compression ratio (UTF-8 bytes over zlib bytes) is at most 2.4, which catches repetition loops.
|
||||
- With hotwords: the clip is decoded once with them and once without, from the same encoder pass.
|
||||
The plain decode must pass the checks above, otherwise nothing was said. The hinted text is used
|
||||
only when it stays within half the plain text's letters of it and mentions every hotword as often
|
||||
as the plain one (a near miss in spelling counts as a mention), so the hint can fix how a spoken
|
||||
wake word is written but never add or drop one. Otherwise the plain text is used.
|
||||
|
||||
`no_speech_prob` is not used: large-v3-turbo reports 0.0 for silence and noise alike.
|
||||
|
||||
Measured (2026-11-11, RTX 5050) with 54 real noise recordings from ESC-50 (typing, mouse, breathing,
|
||||
coughing, laughing, crying, vacuum, ...), each at its own level and at -25 dBFS: of the 108 noise-only
|
||||
clips, 102 never reached the model (the VAD rule above) and the 6 that did (coughs, laughter, crying)
|
||||
all came out empty, with and without the wake word hint. Over 89 mixes of speech with that noise
|
||||
(SNR 20, 10 and 0 dB, plus clean) the mean character error rate was 4.8%. Left over: one
|
||||
"thank you" written for laughter just before a sentence (mean log-probability -0.70, right at the
|
||||
line), two short words lost in loud noise, and one word repeated three times at SNR 0. A stricter
|
||||
line (-0.6) would lose more real words than it saves.
|
||||
|
||||
### TTS
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ from __future__ import annotations
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from voice_worker.audio import InputConverter, peak_normalise
|
||||
from voice_worker.audio import InputConverter, loudness_equalise
|
||||
from voice_worker.protocol import (KIND_AUDIO, KIND_TTS, ProtocolError, decode_audio_frame, encode_audio_frame,
|
||||
encode_tts_frame, decode_tts_frame)
|
||||
|
||||
@@ -104,19 +104,70 @@ def test_odd_byte_counts_are_carried_to_the_next_frame():
|
||||
assert len(a) + len(b) == 10
|
||||
|
||||
|
||||
def _db(x: np.ndarray) -> float:
|
||||
return 20 * np.log10(np.sqrt(np.mean(np.asarray(x, dtype=np.float64) ** 2)) + 1e-12)
|
||||
|
||||
|
||||
def _voice(seconds: float, db: float) -> np.ndarray:
|
||||
"""A stand-in for a voice: a 220 Hz tone with harmonics, at ``db`` dBFS RMS."""
|
||||
t = np.arange(int(16000 * seconds)) / 16000
|
||||
x = sum(np.sin(2 * np.pi * 220 * k * t) / k for k in (1, 2, 3, 4))
|
||||
return (x * 10 ** (db / 20) / np.sqrt(np.mean(x ** 2))).astype(np.float32)
|
||||
|
||||
|
||||
def _mask(n_samples: int, voiced: slice) -> np.ndarray:
|
||||
m = np.zeros(-(-n_samples // 512), dtype=bool)
|
||||
m[voiced] = True
|
||||
return m
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_quiet_clip_is_raised_to_the_target_peak():
|
||||
x = np.full(100, 0.2, dtype=np.float32)
|
||||
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)
|
||||
@pytest.mark.parametrize("db", [-60, -45, -30, -10, -3])
|
||||
def test_every_speaker_level_comes_out_at_the_same_loudness(db):
|
||||
x = _voice(2.0, db)
|
||||
y = loudness_equalise(x)
|
||||
assert y.dtype == np.float32 and len(y) == len(x)
|
||||
assert abs(_db(y) - (-20)) < 1.0
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_loudness_is_measured_on_the_voice_not_the_silence_around_it():
|
||||
x = np.concatenate([np.zeros(16000, dtype=np.float32), _voice(1.0, -40)])
|
||||
y = loudness_equalise(x, _mask(len(x), slice(32, None)))
|
||||
assert abs(_db(y[16000:]) - (-20)) < 1.0
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_loud_click_neither_keeps_a_quiet_voice_quiet_nor_clips():
|
||||
x = _voice(2.0, -50)
|
||||
burst = np.random.default_rng(3).normal(0, 1, 200) * np.exp(-np.arange(200) / 30)
|
||||
x[8000:8200] += (0.95 * burst / np.max(np.abs(burst))).astype(np.float32) # a mouse click, far louder
|
||||
voiced = _mask(len(x), slice(None))
|
||||
voiced[8000 // 512] = False
|
||||
y = loudness_equalise(x, voiced)
|
||||
assert abs(_db(y[:7000]) - (-20)) < 1.0
|
||||
assert np.max(np.abs(y)) < 1.0
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_clipped_shouting_is_brought_down_without_new_clipping():
|
||||
x = np.clip(_voice(2.0, -3) * 4, -1, 1)
|
||||
y = loudness_equalise(x)
|
||||
assert abs(_db(y) - (-20)) < 1.5
|
||||
assert np.max(np.abs(y)) < 1.0
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_near_silence_is_not_boosted_past_the_gain_cap():
|
||||
x = np.full(100, 0.001, dtype=np.float32)
|
||||
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.01)
|
||||
x = _voice(1.0, -90)
|
||||
assert abs(_db(loudness_equalise(x)) - (-90 + 40)) < 1.0
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_loud_clip_is_brought_down_to_the_target_peak():
|
||||
x = np.array([0.0, 1.0, -0.5], dtype=np.float32)
|
||||
assert np.isclose(np.max(np.abs(peak_normalise(x))), 0.9)
|
||||
def test_low_rumble_is_removed_before_levelling():
|
||||
t = np.arange(32000) / 16000
|
||||
rumble = (0.3 * np.sin(2 * np.pi * 30 * t)).astype(np.float32)
|
||||
y = loudness_equalise(_voice(2.0, -40) + rumble)
|
||||
spec = np.abs(np.fft.rfft(y))
|
||||
freqs = np.fft.rfftfreq(len(y), 1 / 16000)
|
||||
assert spec[np.argmin(np.abs(freqs - 30))] < 0.01 * spec[np.argmin(np.abs(freqs - 220))]
|
||||
|
||||
80
tests/test_voice_worker_gate.py
Normal file
80
tests/test_voice_worker_gate.py
Normal file
@@ -0,0 +1,80 @@
|
||||
"""Voice worker: which Whisper transcripts are kept (the hallucination gate), model-free."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from voice_worker.stt_whisper import MIN_MEAN_LOGPROB, choose_with_hint, compression_ratio, keep_transcript
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_confident_transcript_is_kept():
|
||||
assert keep_transcript("자비스, 오늘 서울 날씨 어때?", -0.05)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_low_confidence_is_rejected_even_for_a_short_output():
|
||||
# A three-token hallucination on laughter scored a mean of -0.73 per token; the length
|
||||
# correction used before (len / (len + 1)) lifted it to -0.55 and let it through.
|
||||
assert not keep_transcript("자비스", MIN_MEAN_LOGPROB - 0.03)
|
||||
assert keep_transcript("자비스", MIN_MEAN_LOGPROB + 0.03)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_empty_and_punctuation_only_outputs_are_rejected():
|
||||
assert not keep_transcript("", -0.01)
|
||||
assert not keep_transcript(" . . ", -0.01)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_repetition_loop_is_rejected():
|
||||
loop = "자막 제공 및 " * 12
|
||||
assert compression_ratio(loop) > 2.4
|
||||
assert not keep_transcript(loop, -0.1)
|
||||
assert compression_ratio("오늘 회의는 오후 세 시에 시작해서 다섯 시쯤 끝날 것 같아요.") < 2.4
|
||||
|
||||
|
||||
HINT = ("자비스",)
|
||||
ASK = "지금 두 번째 모니터에 뭐 떠 있는지 알려줄래?"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_voice_like_noise_that_the_hint_turns_into_the_wake_word_gives_nothing():
|
||||
# Laughter, decoded with the wake word as a hint and without it.
|
||||
assert choose_with_hint("자비스", -0.4, "감사합니다.", -1.4, HINT) == ""
|
||||
assert choose_with_hint("자비스", -0.4, "", -0.1, HINT) == ""
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_wake_word_the_hint_adds_to_real_speech_is_dropped():
|
||||
assert choose_with_hint(ASK + " 자비스", -0.1, ASK, -0.1, HINT) == ASK
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_the_hint_may_fix_how_a_spoken_wake_word_is_spelt():
|
||||
fixed = "자비스, 오늘 서울 날씨 어때?"
|
||||
assert choose_with_hint(fixed, -0.1, "자비수, 오늘 서울 날씨 어때?", -0.2, HINT) == fixed
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_wake_word_the_hint_loses_is_kept_from_the_plain_decode():
|
||||
plain = "자비스, 오늘 서울 날씨 어때?"
|
||||
assert choose_with_hint("오늘 서울 날씨 어때?", -0.2, plain, -0.07, HINT) == plain
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_an_unsure_hint_decode_falls_back_to_the_plain_one():
|
||||
assert choose_with_hint(ASK, MIN_MEAN_LOGPROB - 0.2, ASK, -0.1, HINT) == ASK
|
||||
assert choose_with_hint("zzz", -0.1, ASK, -0.1, HINT) == ASK
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_wake_word_matching_ignores_case_spacing_and_punctuation():
|
||||
assert choose_with_hint("Hey Jarvis, lights off.", -0.1, "hey jarvis lights off", -0.1, ("jarvis",)) \
|
||||
== "Hey Jarvis, lights off."
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_spacing_and_punctuation_differences_do_not_change_the_choice():
|
||||
assert choose_with_hint("자비스 오늘 서울 날씨 어때", -0.1, "자비스, 오늘 서울 날씨 어때?", -0.1, HINT) \
|
||||
== "자비스 오늘 서울 날씨 어때"
|
||||
@@ -171,7 +171,10 @@ def test_tts_reply_comes_back_as_the_same_words(engines):
|
||||
if engines[2] is None:
|
||||
pytest.skip("set JARVIS_MELO_PYTHON to the MeloTTS venv python")
|
||||
from voice_worker.protocol import decode_tts_frame, encode_audio_frame
|
||||
text = "오늘 서울은 맑고 기온은 이십삼 도예요. 우산은 필요 없어요."
|
||||
# MeloTTS samples its voice differently each time and blurs some words ("기온은 이십삼" came back
|
||||
# as "비오는 이시찬" in raw, peak-normalised and equalised audio alike), so the sentence is one it
|
||||
# says clearly; this checks the worker's plumbing, not MeloTTS's diction.
|
||||
text = "오늘 서울은 맑고 따뜻해요. 우산은 집에 두고 가셔도 돼요."
|
||||
|
||||
async def body(ws):
|
||||
await ws.send(json.dumps({"type": "config", "partials": False}))
|
||||
@@ -207,3 +210,106 @@ def test_tts_reply_comes_back_as_the_same_words(engines):
|
||||
print(f"tts chunks={len(chunks)} first_chunk_s={first:.2f} heard={heard}")
|
||||
assert [c[1] for c in chunks] == [0, 1]
|
||||
assert cer(text, heard) <= 0.2
|
||||
|
||||
|
||||
def _stream_clip_16k(engines, clip: np.ndarray, stt: dict) -> list[dict]:
|
||||
"""Stream one 16 kHz mono clip as a single speaker (as fast as the socket takes it), end it,
|
||||
and return every final the worker sends."""
|
||||
from voice_worker.protocol import encode_audio_frame
|
||||
pcm = (np.clip(to_48k_stereo(clip), -1, 1) * 32767).astype("<i2").tobytes()
|
||||
step = 48000 * FRAME_MS // 1000 * 2 * 2
|
||||
|
||||
async def body(ws):
|
||||
await ws.send(json.dumps({"type": "config", **stt}))
|
||||
for k in range(0, len(pcm), step):
|
||||
await ws.send(encode_audio_frame("s", pcm[k:k + step]))
|
||||
await ws.send(json.dumps({"type": "speaking_end", "speaker": "s"}))
|
||||
finals = []
|
||||
while True:
|
||||
try:
|
||||
msg = json.loads(await asyncio.wait_for(ws.recv(), 3))
|
||||
except asyncio.TimeoutError:
|
||||
return finals
|
||||
if msg["type"] == "final":
|
||||
finals.append(msg)
|
||||
|
||||
return asyncio.run(with_worker(engines, body))
|
||||
|
||||
|
||||
def _babble(seconds: float, rng) -> np.ndarray:
|
||||
"""Speech-like noise: the kit's Korean clips played backwards and overlaid (voice-like, no words)."""
|
||||
n = int(16000 * seconds)
|
||||
x = np.zeros(n, dtype=np.float32)
|
||||
for name in ("ko_01.wav", "ko_03.wav", "ko_05.wav", "ko_07.wav"):
|
||||
c = load(name)[::-1]
|
||||
x += np.roll(np.tile(c, n // len(c) + 1)[:n], int(rng.integers(0, n)))
|
||||
return x
|
||||
|
||||
|
||||
def _at_db(x: np.ndarray, db: float) -> np.ndarray:
|
||||
return (x * 10 ** (db / 20) / (np.sqrt(np.mean(x.astype(np.float64) ** 2)) + 1e-12)).astype(np.float32)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("level_db", [-40, -25])
|
||||
def test_voice_like_noise_with_the_wake_word_hint_produces_no_text(engines, level_db):
|
||||
"""Backwards babble passes the VAD. Given the wake word as a hint, Whisper used to write the wake
|
||||
word (a phantom call to the bot); without words in the audio no final may carry text."""
|
||||
rng = np.random.default_rng(level_db + 100)
|
||||
finals = _stream_clip_16k(engines, _at_db(_babble(6.0, rng), level_db),
|
||||
{"hotwords": ["자비스"], "partials": False})
|
||||
print(level_db, [(m["text"], m["end_reason"]) for m in finals])
|
||||
assert all(m["text"] == "" for m in finals)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("snr_db", [20, 10])
|
||||
def test_speech_over_voice_like_noise_keeps_the_words(engines, snr_db):
|
||||
rng = np.random.default_rng(snr_db)
|
||||
refs = json.loads((SAMPLES / "refs.json").read_text(encoding="utf-8"))["samples"]
|
||||
errs = []
|
||||
for r in refs[:4]:
|
||||
speech = np.concatenate([np.zeros(8000, np.float32), _at_db(load(r["file"]), -25), np.zeros(8000, np.float32)])
|
||||
noisy = speech + _at_db(_babble(len(speech) / 16000, rng), -25 - snr_db)
|
||||
finals = _stream_clip_16k(engines, noisy, {"hotwords": ["자비스"], "partials": False})
|
||||
text = " ".join(m["text"] for m in finals).strip()
|
||||
errs.append(cer(r["text"], text))
|
||||
print(snr_db, r["file"], round(errs[-1], 2), text)
|
||||
assert np.mean(errs) <= 0.15
|
||||
|
||||
|
||||
@pytest.mark.parametrize("level_db", [-60, -45, -10])
|
||||
def test_quiet_and_loud_speakers_are_both_understood(engines, level_db):
|
||||
refs = json.loads((SAMPLES / "refs.json").read_text(encoding="utf-8"))["samples"]
|
||||
errs = []
|
||||
for r in refs[:4]:
|
||||
x = np.concatenate([np.zeros(8000, np.float32), _at_db(load(r["file"]), level_db), np.zeros(8000, np.float32)])
|
||||
if level_db == -10:
|
||||
x = np.clip(x * 4, -1, 1) # shouting into a hot mic: clipped
|
||||
finals = _stream_clip_16k(engines, x, {"partials": False})
|
||||
errs.append(cer(r["text"], " ".join(m["text"] for m in finals)))
|
||||
print(level_db, [round(e, 2) for e in errs])
|
||||
assert np.mean(errs) <= 0.15
|
||||
|
||||
|
||||
def test_real_noise_recordings_produce_no_text(engines):
|
||||
"""Optional: point JARVIS_NOISE_DIR at noise-only recordings (16 kHz or 44.1 kHz mono/stereo WAV,
|
||||
e.g. ESC-50 keyboard, mouse, breathing, coughing, laughing). None may produce text."""
|
||||
folder = os.environ.get("JARVIS_NOISE_DIR")
|
||||
if not folder:
|
||||
pytest.skip("set JARVIS_NOISE_DIR to a folder of noise-only WAV files")
|
||||
from scipy.signal import resample_poly
|
||||
texts = []
|
||||
for path in sorted(Path(folder).glob("*.wav")):
|
||||
with wave.open(str(path), "rb") as w:
|
||||
x = np.frombuffer(w.readframes(w.getnframes()), dtype="<i2").astype(np.float32) / 32768.0
|
||||
if w.getnchannels() == 2:
|
||||
x = x.reshape(-1, 2).mean(axis=1)
|
||||
rate = w.getframerate()
|
||||
if rate != 16000:
|
||||
from math import gcd
|
||||
g = gcd(rate, 16000)
|
||||
x = resample_poly(x, 16000 // g, rate // g).astype(np.float32)
|
||||
for m in _stream_clip_16k(engines, x, {"hotwords": ["자비스"], "partials": False}):
|
||||
if m["text"]:
|
||||
texts.append((path.name, m["text"]))
|
||||
print(texts)
|
||||
assert texts == []
|
||||
|
||||
@@ -102,6 +102,30 @@ def test_hysteresis_keeps_speech_going_but_never_starts_it():
|
||||
assert len(f.of(Final)) == 1
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_weak_speech_like_noise_never_builds_up_to_an_utterance():
|
||||
"""A breath of real speech probability followed by a long stretch of 'maybe speech' (fans,
|
||||
keyboard, background rumble that the VAD scores around 0.4) is not enough to call the model."""
|
||||
f = Feeder()
|
||||
for _ in range(3):
|
||||
f.feed(0.9, 40)
|
||||
f.feed(0.4, 1000)
|
||||
f.feed(0.0, 500)
|
||||
assert f.events == []
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_finals_mark_which_windows_were_voiced():
|
||||
f = Feeder()
|
||||
f.feed(0.0, 500)
|
||||
f.feed(0.9, 640)
|
||||
f.feed(0.0, 500)
|
||||
fin = f.of(Final)[0]
|
||||
assert len(fin.voiced) == -(-len(fin.audio) // 512)
|
||||
assert 20 <= fin.voiced.sum() <= 21 # 640 ms of 32 ms windows, not aligned to the frames
|
||||
assert not fin.voiced[0] and not fin.voiced[-1] # pre-roll and tail are not voiced
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_speaking_end_closes_the_utterance_at_once():
|
||||
f = Feeder()
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
"""Voice worker end to end over a real local WebSocket, with fake STT/VAD/TTS engines.
|
||||
|
||||
The fake VAD reads speech probability off the amplitude and the fake transcriber names a clip by its
|
||||
peak level, so each speaker's text tells which speaker's audio reached the model.
|
||||
"level": each speaker sends a tone whose amplitude is the level (what the fake VAD reads) and whose
|
||||
pitch is level x 1000 Hz (what the fake transcriber reads, since the worker evens out loudness).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -33,7 +34,7 @@ class LevelTranscriber:
|
||||
|
||||
def transcribe(self, clips, opts):
|
||||
self.calls.append((opts, len(clips)))
|
||||
return [f"level {np.median(np.abs(c)):.1f}" for c in clips]
|
||||
return [f"level {np.argmax(np.abs(np.fft.rfft(c))) * 16000 / len(c) / 1000:.1f}" for c in clips]
|
||||
|
||||
|
||||
class FakeTts:
|
||||
@@ -54,7 +55,9 @@ class FakeTts:
|
||||
|
||||
def pcm(level: float, ms: int, rate=16000, channels=1) -> bytes:
|
||||
n = rate * ms // 1000
|
||||
return (np.full(n * channels, level) * 32767).astype("<i2").tobytes()
|
||||
# 20 ms frames hold whole cycles at these pitches, so frames join without a click.
|
||||
x = level * np.sin(2 * np.pi * level * 1000 * np.arange(n) / rate)
|
||||
return (np.repeat(x, channels) * 32767).astype("<i2").tobytes()
|
||||
|
||||
|
||||
async def with_worker(body, transcriber=None, tts=None, **kw):
|
||||
|
||||
Reference in New Issue
Block a user