feat(voice): real Discord listen+speak loop (STT->echo->TTS)

The bot (dave/bot.mjs) previously only joined the channel and counted audio
frames — it never fed STT or spoke back. Wire the real loop:

- Node bot: buffer each speaker's Opus->PCM utterance until AfterSilence,
  wrap as WAV, POST to the Python voice-turn endpoint, then play the returned
  reply wav into the channel via an AudioPlayer (ffmpeg->Opus). Skips its own
  audio, dedupes overlapping subscriptions, and ignores sub-0.35s noise.
- Python: new `python -m wsai --voice-server` serves /api/voice-turn — decode
  the uploaded utterance, GPU faster-whisper STT, produce a reply (echo of what
  was heard for now), GPU MeloTTS synth, return the reply wav (recognised/reply
  text ride along as X-Heard/X-Reply headers). Both engines pre-warmed; turns
  show in the dashboard feed. MeloTTS.synth() extracted for direct wav reuse.

Echo mode verifies listening+speaking+GPU recognition entirely in Discord; the
Claude brain is the next slice. Verified the endpoint round-trip: utterance wav
-> correct Korean X-Heard/X-Reply + a WAVE reply on device=cuda. 12 tests pass,
node --check clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
EJClaw
2026-08-18 21:59:01 +09:00
parent b9c929a73f
commit 0f94245d5f
4 changed files with 244 additions and 14 deletions

View File

@@ -131,11 +131,13 @@ class MeloTTS:
self.load_ms = info.get("ms")
log.info("melo worker ready in %s ms on %s", self.load_ms, info.get("device"))
async def speak(self, reply: Reply) -> None:
async def synth(self, text: str) -> str:
"""Synthesize `text` to a wav and return its path (no sink). Reusable by
callers that want the wav directly (e.g. the Discord voice bridge)."""
await self._ensure()
self._n += 1
out = str(self.out_dir / f"tts-{self._n:06d}.wav")
req = json.dumps({"text": reply.text, "out": out, "speed": self.speed})
req = json.dumps({"text": text, "out": out, "speed": self.speed})
s = time.monotonic()
async with self._lock:
assert self._proc and self._proc.stdin and self._proc.stdout
@@ -148,7 +150,11 @@ class MeloTTS:
if not res.get("ok"):
raise RuntimeError(f"melo synth failed: {res.get('error')}")
log.debug("synth %d ms (worker %s ms)", int((time.monotonic() - s) * 1000), res.get("ms"))
await self.sink(res["out"], reply)
return res["out"]
async def speak(self, reply: Reply) -> None:
out = await self.synth(reply.text)
await self.sink(out, reply)
async def aclose(self) -> None:
if self._proc is not None and self._proc.returncode is None: