feat(voice): real Discord listen+speak loop (STT->echo->TTS)
The bot (dave/bot.mjs) previously only joined the channel and counted audio frames — it never fed STT or spoke back. Wire the real loop: - Node bot: buffer each speaker's Opus->PCM utterance until AfterSilence, wrap as WAV, POST to the Python voice-turn endpoint, then play the returned reply wav into the channel via an AudioPlayer (ffmpeg->Opus). Skips its own audio, dedupes overlapping subscriptions, and ignores sub-0.35s noise. - Python: new `python -m wsai --voice-server` serves /api/voice-turn — decode the uploaded utterance, GPU faster-whisper STT, produce a reply (echo of what was heard for now), GPU MeloTTS synth, return the reply wav (recognised/reply text ride along as X-Heard/X-Reply headers). Both engines pre-warmed; turns show in the dashboard feed. MeloTTS.synth() extracted for direct wav reuse. Echo mode verifies listening+speaking+GPU recognition entirely in Discord; the Claude brain is the next slice. Verified the endpoint round-trip: utterance wav -> correct Korean X-Heard/X-Reply + a WAVE reply on device=cuda. 12 tests pass, node --check clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -131,11 +131,13 @@ class MeloTTS:
|
||||
self.load_ms = info.get("ms")
|
||||
log.info("melo worker ready in %s ms on %s", self.load_ms, info.get("device"))
|
||||
|
||||
async def speak(self, reply: Reply) -> None:
|
||||
async def synth(self, text: str) -> str:
|
||||
"""Synthesize `text` to a wav and return its path (no sink). Reusable by
|
||||
callers that want the wav directly (e.g. the Discord voice bridge)."""
|
||||
await self._ensure()
|
||||
self._n += 1
|
||||
out = str(self.out_dir / f"tts-{self._n:06d}.wav")
|
||||
req = json.dumps({"text": reply.text, "out": out, "speed": self.speed})
|
||||
req = json.dumps({"text": text, "out": out, "speed": self.speed})
|
||||
s = time.monotonic()
|
||||
async with self._lock:
|
||||
assert self._proc and self._proc.stdin and self._proc.stdout
|
||||
@@ -148,7 +150,11 @@ class MeloTTS:
|
||||
if not res.get("ok"):
|
||||
raise RuntimeError(f"melo synth failed: {res.get('error')}")
|
||||
log.debug("synth %d ms (worker %s ms)", int((time.monotonic() - s) * 1000), res.get("ms"))
|
||||
await self.sink(res["out"], reply)
|
||||
return res["out"]
|
||||
|
||||
async def speak(self, reply: Reply) -> None:
|
||||
out = await self.synth(reply.text)
|
||||
await self.sink(out, reply)
|
||||
|
||||
async def aclose(self) -> None:
|
||||
if self._proc is not None and self._proc.returncode is None:
|
||||
|
||||
Reference in New Issue
Block a user