tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

perf(bridge): stream TTS per sentence to cut voice reply latency

Browse Source 
The /converse turn synthesised the entire reply before any audio played, so
time-to-first-audio grew with reply length. Add a streaming /converse_stream
endpoint that emits the transcript/reply first, then one audio clip per
sentence as each finishes synthesising. The Discord voice layer enqueues each
clip on arrival via the existing FIFO playQueue, so the first sentence starts
speaking while the rest are still being synthesised.

STT and the reply engine still run to completion before the first clip; only
TTS is pipelined. The non-streaming /converse and /text endpoints are
unchanged.

- bridge: language-agnostic sentence splitter (bridge/text_utils.py) + NDJSON
  streaming route
- bot: ndjson() reader + converseStream() client; voice.ts plays clips
  progressively
- tests: splitter unit tests + bot ndjson/converseStream tests

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-13 01:09:39 +09:00
parent 989a4f3e98
commit 5c295420ea
6 changed files with 363 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
bot/src/voice.ts
View File

@@ -23,7 +23,7 @@ import {
} from "@discordjs/voice";
import prism from "prism-media";
import type { VoiceBasedChannel } from "discord.js";
import { converse, decodeWav } from "./bridge.ts";
import { converseStream } from "./bridge.ts";
import { config } from "./config.ts";
const DISCORD_RATE = 48000;
@@ -128,21 +128,27 @@ export class VoiceSession {
const wav = pcm16MonoToWav(mono, DISCORD_RATE);
try {
const result = await converse(wav, this.getBroadcasting?.());
if (result.transcript) {
this.onTurn?.({ user: userId, transcript: result.transcript, reply: result.reply });
}
// Apply any broadcast directive the brain requested (e.g. user said
// "방송 켜줘 / 꺼줘") before playing the reply.
if (result.broadcast_action && this.onBroadcastAction) {
try {
await this.onBroadcastAction(result.broadcast_action);
} catch (e) {
console.error("[voice] broadcast action failed:", e);
}
}
const audio = decodeWav(result.audio_b64);
if (audio) this.play(audio);
// Streaming turn: the brain sends transcript/reply first, then one audio
// clip per sentence as it is synthesised. We enqueue each clip on arrival
// so the first sentence starts playing while the rest are still spoken.
await converseStream(wav, this.getBroadcasting?.(), {
onMeta: async (meta) => {
if (meta.transcript) {
this.onTurn?.({ user: userId, transcript: meta.transcript, reply: meta.reply });
}
// Apply any broadcast directive the brain requested (e.g. user said
// "방송 켜줘 / 꺼줘") before the reply audio plays. The meta line
// always precedes the audio clips, so awaiting here preserves order.
if (meta.broadcast_action && this.onBroadcastAction) {
try {
await this.onBroadcastAction(meta.broadcast_action);
} catch (e) {
console.error("[voice] broadcast action failed:", e);
}
}
},
onAudio: (clip) => this.play(clip),
});
} catch (err) {
console.error("[voice] converse failed:", err);
}