refactor(app): rewrite the M2 Discord voice core in Rust
The user asked for a faster, real app instead of Electron. The app is now a Rust crate (serenity + songbird); the GUI in M5 will be a Tauri 2 shell on the system WebView2. - Same behaviour as the Electron core: one voice channel, per-speaker PCM to the voice worker, speaking_end and speaker_leave, sentence playback with stop and cancel, transcription notice on join, slash command join/leave/stop (create-or-update), registered chat channels with live events, history and sends that never ping, worker process with a fresh token and restart backoff. - songbird 0.6.1 garbles about a third of received frames under DAVE (its decoder keeps the bytes DAVE decryption leaves at the end of the payload; a 20 ms stepped test tone showed repeats, skips and reordering, while sending was exact). songbird now only decrypts; the core reorders packets in a small window (Opus FEC for the last lost frame) and decodes them with libopus itself. - Live test on real Discord with two bots: five sentences heard (CER 0/0/0/5/12.5%), 0.71-0.90 s from end of speech to final, speech read back at 3.4%, sound gone 0.08 s after stop. The release binary uses about 16 MB of memory; a Windows build (MinGW) logs in under Wine. - Spec moved to app/src/discord_core.spec.md; architecture, decisions, roadmap and pc-features updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -29,7 +29,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
|
||||
| `src/jarvis/memory/summariser.spec.md` | Diary summariser prompt contract, hygiene rules (deflection, attribution, topic separation), post-process scrub, and bulk-sweep clean button | Two-layer defence: prompt + deterministic scrub; corrupted summaries poison every downstream consumer |
|
||||
| `src/jarvis/memory/recall_gate.spec.md` | Deterministic skip-enrichment heuristic when the hot window covers a follow-up | Fail-open; language-agnostic via `\w{3,}` + `re.UNICODE`; planner intent always wins |
|
||||
| `src/voice_worker/voice_worker.spec.md` | v2 voice worker: multi-speaker streaming STT (shared Whisper, batched) and sentence TTS over a local WebSocket | Loopback only with an app token; no transcript text in logs; finals before partials; language-agnostic |
|
||||
| `app/src/core/discord_core.spec.md` | v2 Discord voice core in the Electron app: join/leave, per-speaker receive to the voice worker, sentence playback and stop, slash command, registered chat channels (read/send), voice worker process | Hears only the channel it is in and reads only registered channels; join notice on every join; chat is data and sent text never pings; no transcript or chat text in logs |
|
||||
| `app/src/discord_core.spec.md` | v2 Discord voice core in the Rust app: join/leave, per-speaker receive (own reorder window and Opus decoding after songbird decrypts) to the voice worker, sentence playback and stop, slash command, registered chat channels (read/send), voice worker process | Hears only the channel it is in and reads only registered channels; join notice on every join; chat is data and sent text never pings; no transcript or chat text in logs |
|
||||
|
||||
The LLM contexts graph at `docs/llm_contexts.md` maps every LLM call in the app (model, gating, inputs, outputs, limits, flow). Keep it up-to-date at all times: any change that adds, removes, or alters an LLM context (model resolution, timeout, cap, prompt source, gating flag, data-flow edge) must update `docs/llm_contexts.md` in the same PR.
|
||||
|
||||
|
||||
3
app/.gitignore
vendored
3
app/.gitignore
vendored
@@ -1,2 +1 @@
|
||||
node_modules/
|
||||
dist/
|
||||
target/
|
||||
|
||||
4514
app/Cargo.lock
generated
Normal file
4514
app/Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
35
app/Cargo.toml
Normal file
35
app/Cargo.toml
Normal file
@@ -0,0 +1,35 @@
|
||||
[package]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"
|
||||
license = "MIT"
|
||||
publish = false
|
||||
|
||||
[lib]
|
||||
name = "jarvis_app"
|
||||
path = "src/lib.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "jarvis-app"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
serenity = { version = "0.12.5", default-features = false, features = ["client", "gateway", "model", "cache", "http", "voice", "rustls_backend"] }
|
||||
songbird = { version = "0.6.1", features = ["receive", "serenity", "gateway", "driver", "rustls"] }
|
||||
symphonia = { version = "0.5", default-features = false, features = ["wav", "pcm"] }
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
tokio-tungstenite = "0.26"
|
||||
futures-util = "0.3"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
rand = "0.9"
|
||||
base64 = "0.22"
|
||||
async-trait = "0.1"
|
||||
opus2 = "0.4"
|
||||
|
||||
[profile.release]
|
||||
lto = "thin"
|
||||
codegen-units = 4
|
||||
strip = "symbols"
|
||||
2424
app/package-lock.json
generated
2424
app/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -1,29 +0,0 @@
|
||||
{
|
||||
"name": "jarvis-app",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Jarvis v2 Windows app: Discord voice core, voice worker supervisor (Electron + TypeScript)",
|
||||
"type": "module",
|
||||
"main": "dist/main.js",
|
||||
"scripts": {
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"typecheck": "tsc -p tsconfig.json --noEmit && tsc -p test/tsconfig.json",
|
||||
"test": "vitest run --maxWorkers=2",
|
||||
"start": "npm run build && electron .",
|
||||
"start:node": "npm run build && node dist/cli.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"@discordjs/voice": "^0.19.2",
|
||||
"discord.js": "^14.28.1",
|
||||
"opusscript": "^0.0.8",
|
||||
"prism-media": "^1.3.5",
|
||||
"ws": "^8.22.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.19.0",
|
||||
"@types/ws": "^8.18.1",
|
||||
"electron": "^45.3.0",
|
||||
"typescript": "~5.9.3",
|
||||
"vitest": "^3.2.4"
|
||||
}
|
||||
}
|
||||
179
app/src/chat.rs
Normal file
179
app/src/chat.rs
Normal file
@@ -0,0 +1,179 @@
|
||||
//! Registered chat channels: which ones the bot may read or write, and message splitting.
|
||||
|
||||
use crate::settings::ChatChannelSettings;
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Discord's message length limit, in UTF-16 code units as Discord counts.
|
||||
pub const MESSAGE_LIMIT: usize = 2000;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ChatMessage {
|
||||
pub channel_id: u64,
|
||||
pub message_id: u64,
|
||||
pub author_id: u64,
|
||||
pub author_name: String,
|
||||
pub author_is_bot: bool,
|
||||
/// Written by this bot. Live events never carry these; history reads do.
|
||||
pub from_self: bool,
|
||||
/// What the person wrote. Data, never instructions (privacy.md).
|
||||
pub content: String,
|
||||
/// RFC 3339.
|
||||
pub created_at: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ChatAccessError(pub String);
|
||||
|
||||
impl std::fmt::Display for ChatAccessError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str(&self.0)
|
||||
}
|
||||
}
|
||||
impl std::error::Error for ChatAccessError {}
|
||||
|
||||
pub struct ChatChannels {
|
||||
by_id: HashMap<u64, ChatChannelSettings>,
|
||||
}
|
||||
|
||||
impl ChatChannels {
|
||||
pub fn new(channels: &[ChatChannelSettings]) -> Self {
|
||||
Self { by_id: channels.iter().map(|c| (c.id, c.clone())).collect() }
|
||||
}
|
||||
|
||||
/// Some channel is readable, so the bot needs the message intents.
|
||||
pub fn any_readable(&self) -> bool {
|
||||
self.by_id.values().any(|c| c.read)
|
||||
}
|
||||
|
||||
pub fn can_read(&self, id: u64) -> bool {
|
||||
self.by_id.get(&id).is_some_and(|c| c.read)
|
||||
}
|
||||
|
||||
pub fn can_send(&self, id: u64) -> bool {
|
||||
self.by_id.get(&id).is_some_and(|c| c.send)
|
||||
}
|
||||
|
||||
pub fn check_read(&self, id: u64) -> Result<(), ChatAccessError> {
|
||||
match self.by_id.get(&id) {
|
||||
None => Err(ChatAccessError(format!("channel {id} is not a registered chat channel"))),
|
||||
Some(c) if !c.read => Err(ChatAccessError(format!("reading channel {id} is turned off"))),
|
||||
Some(_) => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn check_send(&self, id: u64) -> Result<(), ChatAccessError> {
|
||||
match self.by_id.get(&id) {
|
||||
None => Err(ChatAccessError(format!("channel {id} is not a registered chat channel"))),
|
||||
Some(c) if !c.send => Err(ChatAccessError(format!("sending to channel {id} is turned off"))),
|
||||
Some(_) => Ok(()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Split text into messages of at most `limit` UTF-16 units, breaking at the last newline, else the
|
||||
/// last space, else hard (never inside a character). Nothing is lost except the newline or space a
|
||||
/// break happens at.
|
||||
pub fn split_message(text: &str, limit: usize) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
let mut rest = text;
|
||||
loop {
|
||||
// Byte offset just past the longest prefix within the limit, and whether more follows.
|
||||
let mut units = 0;
|
||||
let mut cut = rest.len();
|
||||
let mut window_end = rest.len(); // the prefix plus one more character, to see a break right at the limit
|
||||
for (i, ch) in rest.char_indices() {
|
||||
if units + ch.len_utf16() > limit {
|
||||
cut = i;
|
||||
window_end = i + ch.len_utf8();
|
||||
break;
|
||||
}
|
||||
units += ch.len_utf16();
|
||||
}
|
||||
if cut == rest.len() {
|
||||
if !rest.is_empty() {
|
||||
out.push(rest.to_owned());
|
||||
}
|
||||
return out;
|
||||
}
|
||||
let window = &rest[..window_end];
|
||||
let brk = window.rfind('\n').filter(|&b| b > 0).or_else(|| window.rfind(' ').filter(|&b| b > 0));
|
||||
match brk {
|
||||
Some(b) => {
|
||||
out.push(rest[..b].to_owned());
|
||||
rest = &rest[b + 1..];
|
||||
}
|
||||
None => {
|
||||
out.push(rest[..cut].to_owned());
|
||||
rest = &rest[cut..];
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Message Content intent bits in the application flags (GATEWAY_MESSAGE_CONTENT and its _LIMITED form).
|
||||
pub fn has_message_content_intent(application_flags: u64) -> bool {
|
||||
application_flags & ((1 << 18) | (1 << 19)) != 0
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn utf16_len(s: &str) -> usize {
|
||||
s.encode_utf16().count()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn short_text_is_left_alone() {
|
||||
assert_eq!(split_message("안녕하세요", MESSAGE_LIMIT), vec!["안녕하세요"]);
|
||||
assert!(split_message("", MESSAGE_LIMIT).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn breaks_at_the_last_newline_then_space_and_loses_only_the_break() {
|
||||
assert_eq!(split_message("aaaa bbbb\ncccc dddd", 12), vec!["aaaa bbbb", "cccc dddd"]);
|
||||
assert_eq!(split_message("aaaa bbbb cccc", 10), vec!["aaaa bbbb", "cccc"]);
|
||||
// A break exactly at the limit is used.
|
||||
assert_eq!(split_message("aaaa bbbb", 4), vec!["aaaa", "bbbb"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hard_cuts_keep_emoji_whole() {
|
||||
let text = format!("{}😀y", "x".repeat(MESSAGE_LIMIT - 1));
|
||||
let parts = split_message(&text, MESSAGE_LIMIT);
|
||||
assert!(parts.iter().all(|p| utf16_len(p) <= MESSAGE_LIMIT));
|
||||
assert_eq!(parts.concat(), text);
|
||||
assert!(parts[1].starts_with('😀'));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_part_fits_for_long_mixed_text() {
|
||||
let text: Vec<String> = (0..900).map(|i| format!("줄 {i} 내용입니다")).collect();
|
||||
let text = text.join("\n");
|
||||
let parts = split_message(&text, MESSAGE_LIMIT);
|
||||
assert!(parts.len() > 1);
|
||||
assert!(parts.iter().all(|p| !p.is_empty() && utf16_len(p) <= MESSAGE_LIMIT));
|
||||
assert_eq!(parts.join("\n"), text);
|
||||
}
|
||||
|
||||
fn ch(id: u64, read: bool, send: bool) -> ChatChannelSettings {
|
||||
ChatChannelSettings { id, name: String::new(), description: String::new(), read, send }
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn each_channel_follows_its_switches_and_strangers_are_refused() {
|
||||
let c = ChatChannels::new(&[ch(1, true, false), ch(2, false, true)]);
|
||||
assert!(c.can_read(1) && !c.can_send(1) && !c.can_read(2) && c.can_send(2));
|
||||
assert!(c.check_send(1).is_err() && c.check_read(2).is_err());
|
||||
assert!(c.check_read(3).unwrap_err().0.contains("not a registered"));
|
||||
assert!(c.any_readable());
|
||||
assert!(!ChatChannels::new(&[ch(2, false, true)]).any_readable());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn message_content_intent_full_or_limited() {
|
||||
assert!(has_message_content_intent(1 << 18));
|
||||
assert!(has_message_content_intent(1 << 19));
|
||||
assert!(!has_message_content_intent((1 << 15) | (1 << 13)));
|
||||
}
|
||||
}
|
||||
@@ -1,23 +0,0 @@
|
||||
/** Run the app's services without Electron (development and tests): node dist/cli.js <data folder> */
|
||||
import { errorLog } from "./core/log.js";
|
||||
import { startServices } from "./services.js";
|
||||
|
||||
const dataDir = process.argv[2] ?? process.env.JARVIS_DATA_DIR;
|
||||
if (!dataDir) {
|
||||
errorLog("usage: node dist/cli.js <data folder> (or set JARVIS_DATA_DIR)");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
startServices(dataDir).then((services) => {
|
||||
let stopping = false;
|
||||
const stop = () => {
|
||||
if (stopping) return;
|
||||
stopping = true;
|
||||
void services.stop().finally(() => process.exit(0));
|
||||
};
|
||||
process.on("SIGINT", stop);
|
||||
process.on("SIGTERM", stop);
|
||||
}, (e: Error) => {
|
||||
errorLog(`could not start: ${e.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,92 +0,0 @@
|
||||
/** Registered chat channels: which ones the bot may read or write, and message splitting. */
|
||||
import type { ChatChannelSettings } from "./settings.js";
|
||||
|
||||
/** Discord's message length limit. */
|
||||
export const MESSAGE_LIMIT = 2000;
|
||||
|
||||
export interface ChatMessage {
|
||||
channelId: string;
|
||||
messageId: string;
|
||||
authorId: string;
|
||||
authorName: string;
|
||||
authorIsBot: boolean;
|
||||
/** Written by this bot. Live events never carry these; history reads do. */
|
||||
fromSelf: boolean;
|
||||
/** What the person wrote. Data, never instructions (privacy.md). */
|
||||
content: string;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export class ChatAccessError extends Error {}
|
||||
|
||||
export class ChatChannels {
|
||||
private byId: Map<string, ChatChannelSettings>;
|
||||
|
||||
constructor(channels: ChatChannelSettings[]) {
|
||||
this.byId = new Map(channels.map((c) => [c.id, c]));
|
||||
}
|
||||
|
||||
get(id: string): ChatChannelSettings | undefined {
|
||||
return this.byId.get(id);
|
||||
}
|
||||
|
||||
list(): ChatChannelSettings[] {
|
||||
return [...this.byId.values()];
|
||||
}
|
||||
|
||||
/** Some channel is readable, so the bot needs the message intents. */
|
||||
get anyReadable(): boolean {
|
||||
return this.list().some((c) => c.read);
|
||||
}
|
||||
|
||||
canRead(id: string): boolean {
|
||||
return this.byId.get(id)?.read === true;
|
||||
}
|
||||
|
||||
canSend(id: string): boolean {
|
||||
return this.byId.get(id)?.send === true;
|
||||
}
|
||||
|
||||
assertRead(id: string): void {
|
||||
if (!this.byId.has(id)) throw new ChatAccessError(`channel ${id} is not a registered chat channel`);
|
||||
if (!this.canRead(id)) throw new ChatAccessError(`reading channel ${id} is turned off`);
|
||||
}
|
||||
|
||||
assertSend(id: string): void {
|
||||
if (!this.byId.has(id)) throw new ChatAccessError(`channel ${id} is not a registered chat channel`);
|
||||
if (!this.canSend(id)) throw new ChatAccessError(`sending to channel ${id} is turned off`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Split text into messages of at most `limit` characters (UTF-16 units, as Discord counts),
|
||||
* breaking at the last newline, else the last space, else hard. Nothing is lost except the
|
||||
* newline or space a break happens at.
|
||||
*/
|
||||
export function splitMessage(text: string, limit = MESSAGE_LIMIT): string[] {
|
||||
const out: string[] = [];
|
||||
let rest = text;
|
||||
while (rest.length > limit) {
|
||||
const window = rest.slice(0, limit + 1);
|
||||
let cut = window.lastIndexOf("\n");
|
||||
if (cut <= 0) cut = window.lastIndexOf(" ");
|
||||
if (cut <= 0) {
|
||||
cut = limit;
|
||||
// Do not split a surrogate pair.
|
||||
const code = rest.charCodeAt(cut - 1);
|
||||
if (code >= 0xd800 && code <= 0xdbff) cut--;
|
||||
out.push(rest.slice(0, cut));
|
||||
rest = rest.slice(cut);
|
||||
} else {
|
||||
out.push(rest.slice(0, cut));
|
||||
rest = rest.slice(cut + 1);
|
||||
}
|
||||
}
|
||||
if (rest.length) out.push(rest);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Intent bits in the application flags (GATEWAY_MESSAGE_CONTENT and its _LIMITED form). */
|
||||
export function hasMessageContentIntent(applicationFlags: number): boolean {
|
||||
return (applicationFlags & ((1 << 18) | (1 << 19))) !== 0;
|
||||
}
|
||||
@@ -1,502 +0,0 @@
|
||||
/**
|
||||
* The Discord voice core (app/src/core/discord_core.spec.md).
|
||||
*
|
||||
* One bot, one server, one voice channel at a time. Hears each speaker separately and streams their
|
||||
* PCM to the voice worker, speaks text through the worker's TTS, and reads and writes the registered
|
||||
* chat channels. It has no assistant logic: it reports what was heard and does what it is told.
|
||||
*/
|
||||
import { EventEmitter } from "node:events";
|
||||
import { Readable } from "node:stream";
|
||||
import {
|
||||
ApplicationIntegrationType,
|
||||
Client,
|
||||
Events,
|
||||
GatewayIntentBits,
|
||||
InteractionContextType,
|
||||
MessageFlags,
|
||||
REST,
|
||||
Routes,
|
||||
SlashCommandBuilder,
|
||||
type ChatInputCommandInteraction,
|
||||
type GuildMember,
|
||||
type Message,
|
||||
type VoiceBasedChannel,
|
||||
type VoiceState,
|
||||
} from "discord.js";
|
||||
import {
|
||||
AudioPlayerStatus,
|
||||
createAudioPlayer,
|
||||
createAudioResource,
|
||||
EndBehaviorType,
|
||||
entersState,
|
||||
joinVoiceChannel,
|
||||
NoSubscriberBehavior,
|
||||
StreamType,
|
||||
VoiceConnectionStatus,
|
||||
type AudioPlayer,
|
||||
type AudioResource,
|
||||
type VoiceConnection,
|
||||
} from "@discordjs/voice";
|
||||
import OpusScript from "opusscript";
|
||||
import { ChatAccessError, ChatChannels, hasMessageContentIntent, splitMessage, type ChatMessage } from "./chat.js";
|
||||
import { debugLog, errorLog } from "./log.js";
|
||||
import { Playback, type PlaybackEnd } from "./playback.js";
|
||||
import type { Settings } from "./settings.js";
|
||||
import { SpeakerGate } from "./speakers.js";
|
||||
import type { Final, VoiceWorkerClient } from "./workerClient.js";
|
||||
|
||||
/** Discord voice is 48 kHz stereo PCM after Opus decoding; the worker is told the same in hello. */
|
||||
export const DISCORD_PCM = { rate: 48000, channels: 2 } as const;
|
||||
|
||||
export interface HeardSpeech {
|
||||
speakerId: string;
|
||||
speakerName: string;
|
||||
utt: number;
|
||||
}
|
||||
|
||||
export interface HeardFinal extends HeardSpeech {
|
||||
text: string;
|
||||
endReason: string;
|
||||
audioMs: number;
|
||||
waitMs: number;
|
||||
sttMs: number;
|
||||
}
|
||||
|
||||
export interface CoreEvents {
|
||||
ready: [{ userId: string; tag: string }];
|
||||
joined: [{ guildId: string; channelId: string }];
|
||||
left: [{ guildId: string; channelId: string; reason: string }];
|
||||
speech_start: [HeardSpeech];
|
||||
partial: [HeardSpeech & { text: string }];
|
||||
final: [HeardFinal];
|
||||
speak_start: [string];
|
||||
speak_end: [PlaybackEnd];
|
||||
chat: [ChatMessage];
|
||||
}
|
||||
|
||||
interface SpeakerPipe {
|
||||
stream: Readable;
|
||||
decoder: OpusScript;
|
||||
}
|
||||
|
||||
/** A live voice connection: per-speaker receive and the playback queue. */
|
||||
class VoiceSession {
|
||||
readonly connection: VoiceConnection;
|
||||
readonly player: AudioPlayer;
|
||||
readonly playback: Playback;
|
||||
private current: AudioResource | null = null;
|
||||
private pipes = new Map<string, SpeakerPipe>();
|
||||
private destroyed = false;
|
||||
badPackets = 0;
|
||||
|
||||
constructor(
|
||||
readonly guildId: string,
|
||||
public channelId: string,
|
||||
channel: VoiceBasedChannel,
|
||||
private readonly worker: VoiceWorkerClient,
|
||||
private readonly gate: SpeakerGate,
|
||||
private readonly shouldHear: (userId: string) => boolean,
|
||||
) {
|
||||
this.connection = joinVoiceChannel({
|
||||
channelId: channel.id,
|
||||
guildId: channel.guild.id,
|
||||
adapterCreator: channel.guild.voiceAdapterCreator,
|
||||
selfDeaf: false,
|
||||
selfMute: false,
|
||||
});
|
||||
this.player = createAudioPlayer({ behaviors: { noSubscriber: NoSubscriberBehavior.Pause } });
|
||||
this.connection.subscribe(this.player);
|
||||
this.playback = new Playback(worker, {
|
||||
play: (pcm) => {
|
||||
const resource = createAudioResource(Readable.from([pcm]), { inputType: StreamType.Raw });
|
||||
this.current = resource;
|
||||
this.player.play(resource);
|
||||
},
|
||||
stop: () => {
|
||||
this.current = null;
|
||||
this.player.stop(true);
|
||||
},
|
||||
});
|
||||
this.player.on("stateChange", (oldState, newState) => {
|
||||
// Only the end of the clip we started counts; a stop() already moved on.
|
||||
if (oldState.status !== AudioPlayerStatus.Idle && newState.status === AudioPlayerStatus.Idle
|
||||
&& "resource" in oldState && oldState.resource === this.current) {
|
||||
this.current = null;
|
||||
this.playback.clipEnded();
|
||||
}
|
||||
});
|
||||
this.player.on("error", (e) => errorLog(`playback error: ${e.message}`, "voice"));
|
||||
const speaking = this.connection.receiver.speaking;
|
||||
speaking.on("start", (userId) => this.onSpeakingStart(userId));
|
||||
speaking.on("end", (userId) => this.onSpeakingEnd(userId));
|
||||
}
|
||||
|
||||
async ready(timeoutMs = 20_000): Promise<void> {
|
||||
await entersState(this.connection, VoiceConnectionStatus.Ready, timeoutMs);
|
||||
}
|
||||
|
||||
private onSpeakingStart(userId: string): void {
|
||||
if (this.destroyed || !this.shouldHear(userId)) return;
|
||||
if (!this.gate.start(userId)) {
|
||||
debugLog(`speaker ${userId} not heard: ${this.gate.count} already speaking`, "voice");
|
||||
return;
|
||||
}
|
||||
this.subscribe(userId);
|
||||
}
|
||||
|
||||
private onSpeakingEnd(userId: string): void {
|
||||
if (this.gate.end(userId)) this.worker.speakingEnd(userId);
|
||||
}
|
||||
|
||||
/** One long-lived Opus subscription per speaker; PCM goes to the worker only while they are heard. */
|
||||
private subscribe(userId: string): void {
|
||||
if (this.pipes.has(userId)) return;
|
||||
const stream = this.connection.receiver.subscribe(userId, { end: { behavior: EndBehaviorType.Manual } });
|
||||
const decoder = new OpusScript(DISCORD_PCM.rate, DISCORD_PCM.channels, OpusScript.Application.VOIP);
|
||||
const pipe: SpeakerPipe = { stream, decoder };
|
||||
this.pipes.set(userId, pipe);
|
||||
stream.on("data", (packet: Buffer) => {
|
||||
if (!this.gate.isActive(userId)) return;
|
||||
let pcm: Buffer;
|
||||
try {
|
||||
pcm = decoder.decode(packet);
|
||||
} catch {
|
||||
this.badPackets++;
|
||||
return;
|
||||
}
|
||||
this.worker.sendAudio(userId, pcm);
|
||||
});
|
||||
stream.on("error", (e) => debugLog(`receive stream for ${userId} failed: ${e.message}`, "voice"));
|
||||
stream.once("close", () => {
|
||||
if (this.pipes.get(userId) !== pipe) return;
|
||||
this.pipes.delete(userId);
|
||||
decoder.delete();
|
||||
// A packet that failed to decrypt destroys the stream; pick the speaker up again at once.
|
||||
if (!this.destroyed && this.gate.isActive(userId)) this.subscribe(userId);
|
||||
});
|
||||
}
|
||||
|
||||
/** The person left the channel: finish their utterance and free their place in the worker. */
|
||||
speakerLeft(userId: string): void {
|
||||
const wasHeard = this.gate.end(userId);
|
||||
const pipe = this.pipes.get(userId);
|
||||
if (pipe) {
|
||||
this.pipes.delete(userId);
|
||||
pipe.stream.destroy();
|
||||
pipe.decoder.delete();
|
||||
}
|
||||
if (wasHeard || pipe) this.worker.speakerLeave(userId);
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
if (this.destroyed) return;
|
||||
this.destroyed = true;
|
||||
this.playback.stop();
|
||||
for (const [userId, pipe] of this.pipes) {
|
||||
pipe.stream.destroy();
|
||||
pipe.decoder.delete();
|
||||
this.worker.speakerLeave(userId);
|
||||
}
|
||||
this.pipes.clear();
|
||||
this.gate.clear();
|
||||
this.player.stop(true);
|
||||
if (this.connection.state.status !== VoiceConnectionStatus.Destroyed) this.connection.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
export class DiscordCore extends EventEmitter<CoreEvents> {
|
||||
private _client: Client | null = null;
|
||||
private session: VoiceSession | null = null;
|
||||
private gate: SpeakerGate;
|
||||
private chat: ChatChannels;
|
||||
private names = new Map<string, string>();
|
||||
/** The application has the Message Content intent; without it chat channels cannot be read. */
|
||||
chatContent = false;
|
||||
|
||||
constructor(private readonly settings: Settings, private readonly token: string,
|
||||
private readonly worker: VoiceWorkerClient) {
|
||||
super();
|
||||
this.gate = new SpeakerGate(settings.voice.maxSpeakers);
|
||||
this.chat = new ChatChannels(settings.chatChannels);
|
||||
worker.on("speech_start", (m) => this.emit("speech_start", this.heard(m.speaker, m.utt)));
|
||||
worker.on("partial", (m) => this.emit("partial", { ...this.heard(m.speaker, m.utt), text: m.text }));
|
||||
worker.on("final", (m) => this.onFinal(m));
|
||||
worker.on("tts_audio", (f) => this.session?.playback.audio(f.id, f.seq, f.pcm));
|
||||
worker.on("tts_done", (d) => this.session?.playback.done(d));
|
||||
}
|
||||
|
||||
/** The discord.js client; exists once start() has begun (intents are chosen at login). */
|
||||
get client(): Client {
|
||||
if (!this._client) throw new Error("the Discord core has not been started");
|
||||
return this._client;
|
||||
}
|
||||
|
||||
/** Log in, register the slash command, and join the auto-join channel if one is set. */
|
||||
async start(): Promise<void> {
|
||||
const rest = new REST({ version: "10" }).setToken(this.token);
|
||||
const app = (await rest.get(Routes.currentApplication())) as { id: string; flags?: number };
|
||||
this.chatContent = hasMessageContentIntent(app.flags ?? 0);
|
||||
const intents = [GatewayIntentBits.Guilds, GatewayIntentBits.GuildVoiceStates];
|
||||
if (this.chat.anyReadable) {
|
||||
intents.push(GatewayIntentBits.GuildMessages);
|
||||
if (this.chatContent) intents.push(GatewayIntentBits.MessageContent);
|
||||
else errorLog("Message Content Intent is off for this bot; chat channels cannot be read. "
|
||||
+ "Turn it on in the Discord Developer Portal (Bot > Privileged Gateway Intents).", "discord");
|
||||
}
|
||||
const client = new Client({ intents });
|
||||
this._client = client;
|
||||
client.on(Events.InteractionCreate, (i) => {
|
||||
if (i.isChatInputCommand()) void this.onCommand(i);
|
||||
});
|
||||
client.on(Events.VoiceStateUpdate, (o, n) => this.onVoiceState(o, n));
|
||||
client.on(Events.MessageCreate, (m) => this.onMessage(m));
|
||||
client.on(Events.Error, (e) => errorLog(`gateway error: ${e.message}`, "discord"));
|
||||
const ready = new Promise<void>((resolve) => client.once(Events.ClientReady, () => resolve()));
|
||||
await client.login(this.token);
|
||||
await ready;
|
||||
debugLog(`logged in as ${client.user!.id} chatContent=${this.chatContent}`, "discord");
|
||||
this.emit("ready", { userId: client.user!.id, tag: client.user!.tag });
|
||||
try {
|
||||
await this.registerCommand(rest, app.id);
|
||||
} catch (e) {
|
||||
// Voice and chat still work; the command can be registered on the next start.
|
||||
errorLog(`could not register the slash command: ${(e as Error).message}`, "discord");
|
||||
}
|
||||
const auto = this.settings.discord.voiceChannelId;
|
||||
if (auto) {
|
||||
try {
|
||||
await this.join(auto);
|
||||
} catch (e) {
|
||||
errorLog(`auto-join ${auto} failed: ${(e as Error).message}`, "voice");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async registerCommand(rest: REST, appId: string): Promise<void> {
|
||||
const guildId = this.settings.discord.guildId;
|
||||
if (!guildId) {
|
||||
errorLog("discord.guildId is not set; the slash command is not registered", "discord");
|
||||
return;
|
||||
}
|
||||
const cmd = new SlashCommandBuilder()
|
||||
.setName(this.settings.discord.commandName)
|
||||
.setDescription("자비스 음성 비서")
|
||||
.setContexts(InteractionContextType.Guild)
|
||||
.setIntegrationTypes(ApplicationIntegrationType.GuildInstall)
|
||||
.addSubcommand((s) => s.setName("join").setDescription("내가 있는 음성 채널로 자비스를 부릅니다"))
|
||||
.addSubcommand((s) => s.setName("leave").setDescription("자비스를 음성 채널에서 내보냅니다"))
|
||||
.addSubcommand((s) => s.setName("stop").setDescription("자비스가 하던 말을 멈춥니다"));
|
||||
// POST creates or updates this one command by name and leaves the app's other commands alone.
|
||||
await rest.post(Routes.applicationGuildCommands(appId, guildId), { body: cmd.toJSON() });
|
||||
debugLog(`slash command /${cmd.name} registered in ${guildId}`, "discord");
|
||||
}
|
||||
|
||||
// --- voice ------------------------------------------------------------------------------------
|
||||
|
||||
get voiceChannelId(): string | null {
|
||||
return this.session?.channelId ?? null;
|
||||
}
|
||||
|
||||
/** Join a voice channel of the configured server (leaving the current one) and post the notice. */
|
||||
async join(channelId: string): Promise<void> {
|
||||
const channel = await this.client.channels.fetch(channelId);
|
||||
if (!channel?.isVoiceBased()) throw new Error(`channel ${channelId} is not a voice channel`);
|
||||
const guildId = this.settings.discord.guildId;
|
||||
if (guildId && channel.guild.id !== guildId) throw new Error(`channel ${channelId} is not in the configured server`);
|
||||
if (this.session?.channelId === channel.id) return;
|
||||
this.leave("moved");
|
||||
const session = new VoiceSession(channel.guild.id, channel.id, channel, this.worker, this.gate,
|
||||
(id) => this.shouldHear(channel.guild.id, id));
|
||||
this.session = session;
|
||||
session.playback.on("start", (id) => this.emit("speak_start", id));
|
||||
session.playback.on("end", (e) => this.emit("speak_end", e));
|
||||
session.connection.on(VoiceConnectionStatus.Disconnected, async () => {
|
||||
try {
|
||||
// Moved to another channel or a brief network drop: the library reconnects on its own.
|
||||
await Promise.race([
|
||||
entersState(session.connection, VoiceConnectionStatus.Signalling, 5000),
|
||||
entersState(session.connection, VoiceConnectionStatus.Connecting, 5000),
|
||||
]);
|
||||
} catch {
|
||||
if (this.session === session) this.leave("disconnected");
|
||||
}
|
||||
});
|
||||
try {
|
||||
await session.ready();
|
||||
} catch (e) {
|
||||
if (this.session === session) this.leave("connect_failed");
|
||||
throw new Error(`could not connect to voice channel ${channelId}: ${(e as Error).message}`);
|
||||
}
|
||||
debugLog(`joined voice ${channel.id}`, "voice");
|
||||
this.emit("joined", { guildId: channel.guild.id, channelId: channel.id });
|
||||
try {
|
||||
await channel.send({ content: this.settings.voice.joinNotice, allowedMentions: { parse: [] } });
|
||||
} catch (e) {
|
||||
errorLog(`could not post the transcription notice in ${channel.id}: ${(e as Error).message}`, "voice");
|
||||
}
|
||||
}
|
||||
|
||||
/** Leave the voice channel. False if not in one. */
|
||||
leave(reason = "command"): boolean {
|
||||
const s = this.session;
|
||||
if (!s) return false;
|
||||
this.session = null;
|
||||
s.destroy();
|
||||
debugLog(`left voice ${s.channelId} reason=${reason} bad_packets=${s.badPackets}`, "voice");
|
||||
this.emit("left", { guildId: s.guildId, channelId: s.channelId, reason });
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Speak text in the voice channel. Returns the request id, or null when not in a voice channel. */
|
||||
say(text: string, more = false): string | null {
|
||||
return this.session ? this.session.playback.say(text, more) : null;
|
||||
}
|
||||
|
||||
/** More text for a request started with say(..., true). */
|
||||
continueText(id: string, text: string, more = false): void {
|
||||
this.session?.playback.continueText(id, text, more);
|
||||
}
|
||||
|
||||
/** Stop speaking at once and drop everything queued. Returns how many requests were stopped. */
|
||||
stop(): number {
|
||||
return this.session?.playback.stop() ?? 0;
|
||||
}
|
||||
|
||||
get speaking(): boolean {
|
||||
return this.session?.playback.busy ?? false;
|
||||
}
|
||||
|
||||
private shouldHear(guildId: string, userId: string): boolean {
|
||||
if (userId === this.client.user?.id) return false;
|
||||
if (this.settings.voice.listenToBots) return true;
|
||||
const user = this.client.users.cache.get(userId) ?? this.client.guilds.cache.get(guildId)?.members.cache.get(userId)?.user;
|
||||
// Unknown users are people: the voice state that put them in the channel would have cached a bot.
|
||||
return !user?.bot;
|
||||
}
|
||||
|
||||
private onVoiceState(oldState: VoiceState, newState: VoiceState): void {
|
||||
const s = this.session;
|
||||
if (!s || oldState.guild.id !== s.guildId) return;
|
||||
const userId = newState.id;
|
||||
if (userId === this.client.user?.id) {
|
||||
// Moved by someone else: follow the new channel id.
|
||||
if (newState.channelId && newState.channelId !== s.channelId) s.channelId = newState.channelId;
|
||||
return;
|
||||
}
|
||||
if (newState.member) this.names.set(userId, displayName(newState.member));
|
||||
if (oldState.channelId === s.channelId && newState.channelId !== s.channelId) s.speakerLeft(userId);
|
||||
}
|
||||
|
||||
private heard(speakerId: string, utt: number): HeardSpeech {
|
||||
return { speakerId, speakerName: this.nameOf(speakerId), utt };
|
||||
}
|
||||
|
||||
private nameOf(userId: string): string {
|
||||
const cached = this.names.get(userId);
|
||||
if (cached) return cached;
|
||||
const guildId = this.session?.guildId ?? this.settings.discord.guildId;
|
||||
const member = guildId ? this.client.guilds.cache.get(guildId)?.members.cache.get(userId) : undefined;
|
||||
const name = member ? displayName(member) : this.client.users.cache.get(userId)?.displayName ?? userId;
|
||||
this.names.set(userId, name);
|
||||
return name;
|
||||
}
|
||||
|
||||
private onFinal(m: Final): void {
|
||||
debugLog(`final speaker=${m.speaker} utt=${m.utt} reason=${m.end_reason} chars=${m.text.length} `
|
||||
+ `audio_ms=${m.audio_ms} wait_ms=${m.wait_ms}`, "voice");
|
||||
this.emit("final", {
|
||||
...this.heard(m.speaker, m.utt), text: m.text, endReason: m.end_reason, audioMs: m.audio_ms,
|
||||
waitMs: m.wait_ms, sttMs: m.stt_ms,
|
||||
});
|
||||
}
|
||||
|
||||
// --- slash command ----------------------------------------------------------------------------
|
||||
|
||||
private async onCommand(i: ChatInputCommandInteraction): Promise<void> {
|
||||
if (i.commandName !== this.settings.discord.commandName) return;
|
||||
const guildId = this.settings.discord.guildId;
|
||||
if (!i.inGuild() || (guildId && i.guildId !== guildId)) return;
|
||||
const flags = this.settings.discord.ephemeralReplies ? MessageFlags.Ephemeral : undefined;
|
||||
const sub = i.options.getSubcommand();
|
||||
debugLog(`command ${sub} from ${i.user.id}`, "discord");
|
||||
try {
|
||||
if (sub === "join") {
|
||||
const member = i.member as GuildMember | null;
|
||||
const channelId = member?.voice?.channelId
|
||||
?? i.guild?.voiceStates.cache.get(i.user.id)?.channelId ?? null;
|
||||
if (!channelId) {
|
||||
await i.reply({ content: "먼저 음성 채널에 들어간 뒤 다시 불러 주세요.", flags });
|
||||
return;
|
||||
}
|
||||
await i.deferReply({ flags });
|
||||
await this.join(channelId);
|
||||
await i.editReply("음성 채널에 들어왔습니다.");
|
||||
} else if (sub === "leave") {
|
||||
await i.reply({ content: this.leave() ? "음성 채널에서 나갔습니다." : "음성 채널에 있지 않습니다.", flags });
|
||||
} else if (sub === "stop") {
|
||||
await i.reply({ content: this.stop() ? "멈췄습니다." : "지금은 말하고 있지 않습니다.", flags });
|
||||
}
|
||||
} catch (e) {
|
||||
errorLog(`command ${sub} failed: ${(e as Error).message}`, "discord");
|
||||
const content = "처리하지 못했습니다. 잠시 뒤 다시 시도해 주세요.";
|
||||
if (i.deferred || i.replied) await i.editReply(content).catch(() => {});
|
||||
else await i.reply({ content, flags }).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
// --- chat channels ----------------------------------------------------------------------------
|
||||
|
||||
private toChat(m: Message): ChatMessage {
|
||||
return {
|
||||
channelId: m.channelId,
|
||||
messageId: m.id,
|
||||
authorId: m.author.id,
|
||||
authorName: m.member ? displayName(m.member) : m.author.displayName,
|
||||
authorIsBot: m.author.bot,
|
||||
fromSelf: m.author.id === this.client.user?.id,
|
||||
content: m.content,
|
||||
createdAt: m.createdAt.toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
private onMessage(m: Message): void {
|
||||
if (!this.chatContent || !this.chat.canRead(m.channelId) || m.author.id === this.client.user?.id) return;
|
||||
debugLog(`chat message in ${m.channelId} from ${m.author.id} chars=${m.content.length}`, "chat");
|
||||
this.emit("chat", this.toChat(m));
|
||||
}
|
||||
|
||||
/** The latest `limit` messages of a registered channel, oldest first. */
|
||||
async readChat(channelId: string, limit = 20): Promise<ChatMessage[]> {
|
||||
this.chat.assertRead(channelId);
|
||||
if (!this.chatContent) throw new ChatAccessError("the bot's Message Content Intent is off");
|
||||
const channel = await this.client.channels.fetch(channelId);
|
||||
if (!channel?.isTextBased()) throw new ChatAccessError(`channel ${channelId} is not a text channel`);
|
||||
const messages = await channel.messages.fetch({ limit: Math.max(1, Math.min(100, limit)) });
|
||||
return [...messages.values()].sort((a, b) => a.createdTimestamp - b.createdTimestamp).map((m) => this.toChat(m));
|
||||
}
|
||||
|
||||
/** Send text to a registered channel, split at Discord's length limit. Returns the message ids. */
|
||||
async sendChat(channelId: string, text: string): Promise<string[]> {
|
||||
this.chat.assertSend(channelId);
|
||||
if (!text.trim()) throw new ChatAccessError("nothing to send");
|
||||
const channel = await this.client.channels.fetch(channelId);
|
||||
if (!channel?.isSendable()) throw new ChatAccessError(`channel ${channelId} is not a channel the bot can send to`);
|
||||
const ids: string[] = [];
|
||||
for (const part of splitMessage(text)) {
|
||||
// Never ping anyone from text the assistant wrote.
|
||||
const sent = await channel.send({ content: part, allowedMentions: { parse: [] } });
|
||||
ids.push(sent.id);
|
||||
}
|
||||
debugLog(`chat sent to ${channelId} parts=${ids.length} chars=${text.length}`, "chat");
|
||||
return ids;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.leave("shutdown");
|
||||
await this._client?.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
function displayName(member: GuildMember): string {
|
||||
return member.displayName || member.user.globalName || member.user.username;
|
||||
}
|
||||
@@ -1,163 +0,0 @@
|
||||
# Discord voice core spec (v2 M2)
|
||||
|
||||
The Discord voice core is the part of the Windows app (Electron main process, TypeScript) that
|
||||
connects the bot to Discord: it joins a voice channel, hears each person separately, streams their
|
||||
audio to the voice worker for transcription, plays the worker's speech back into the channel, and reads
|
||||
and writes the registered chat channels. It has no assistant logic. It reports what it heard and does
|
||||
what it is told; the brain (M3) and call detection and response policies (M4) sit on top of it.
|
||||
Design background: `docs/v2/architecture.md`, `docs/v2/conversation.md`, `docs/v2/privacy.md`. The
|
||||
worker side of the link is `src/voice_worker/voice_worker.spec.md`.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Only where the bot is.** The bot hears only the voice channel it is in and reads only the chat
|
||||
channels registered in settings with reading on. Every join posts the transcription notice in the
|
||||
voice channel's chat.
|
||||
- **Chat is data.** Messages read from chat channels are handed on as data, never acted on as
|
||||
commands. Text the bot sends never pings anyone (`allowedMentions: { parse: [] }`).
|
||||
- **No text in logs.** Logs carry ids, sizes, counts and timings, never what anyone said or wrote, or
|
||||
what the bot says. Debug logs (`debugLog`, on with `JARVIS_DEBUG=1`) go at joins and leaves, command
|
||||
use, finals, speech requests and stops, chat events and sends, and worker connection changes.
|
||||
- **Language-agnostic.** Nothing in the core matches words. Wake words are only passed to the STT as a
|
||||
hint until call detection (M4) uses them.
|
||||
- **Testable without Discord.** The pieces with rules (playback queue, speaker limit, chat access and
|
||||
splitting, settings, worker link, worker process) are plain classes tested with fakes; the Discord
|
||||
glue is tested on real Discord by an opt-in live test.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
app/
|
||||
├── package.json # Electron + TypeScript; npm test, npm run build, npm start
|
||||
├── src/main.ts # Electron main: starts the services, stops them on quit (no window until M5)
|
||||
├── src/cli.ts # The same services under plain Node: node dist/cli.js <data folder>
|
||||
├── src/services.ts # Settings -> voice worker process -> worker client -> Discord core
|
||||
└── src/core/
|
||||
├── discordCore.ts # Discord client, voice session, slash command, chat channels
|
||||
├── playback.ts # Speech queue (TTS requests -> sentence clips -> player)
|
||||
├── speakers.ts # Who is transcribed right now (max speakers)
|
||||
├── chat.ts # Chat channel access rules and message splitting
|
||||
├── workerClient.ts # WebSocket client for the voice worker
|
||||
├── workerProcess.ts # Starts and restarts the Python voice worker
|
||||
├── frames.ts # Binary frame layout shared with voice_worker/protocol.py
|
||||
├── settings.ts # settings.json loading and validation
|
||||
└── log.ts # debugLog / errorLog
|
||||
```
|
||||
|
||||
## Starting
|
||||
|
||||
`startServices(dataDir)` reads `<dataDir>/config/settings.json`, starts the voice worker process,
|
||||
connects to it once it prints `READY`, and starts the Discord core. Until the setup wizard (M5) stores
|
||||
it encrypted, the bot token comes from `JARVIS_DISCORD_TOKEN` and the data folder from
|
||||
`JARVIS_DATA_DIR` (Electron) or the first argument (`cli.js`). The Electron entry holds a single
|
||||
instance lock and stops every service before it quits.
|
||||
|
||||
## Settings
|
||||
|
||||
`<data folder>/config/settings.json`. Every key is optional; missing keys take the defaults. A bad
|
||||
value stops the app with a message naming the key. Unknown keys are ignored.
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `discord.guildId` | `null` | The one server. The slash command is registered there; commands and channels from other servers are ignored. |
|
||||
| `discord.voiceChannelId` | `null` | Join this voice channel on start ("자동 입장"). |
|
||||
| `discord.commandName` | `"자비스"` | Slash command name (Discord rule: 1-32 lowercase letters, digits, `-`, `_`). |
|
||||
| `discord.ephemeralReplies` | `true` | Command replies only the caller sees. |
|
||||
| `voice.maxSpeakers` | `10` | Most people transcribed at once, 1-10. |
|
||||
| `voice.listenToBots` | `false` | Also transcribe other bots (music bots are ignored by default). |
|
||||
| `voice.joinNotice` | "자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다." | Posted in the voice channel's chat on every join. Must not be empty. |
|
||||
| `voice.wakeWords` | `["자비스"]` | Sent to the worker as STT `hotwords`. |
|
||||
| `voice.stt` | `{}` | Other worker STT settings (`partials`, `end_silence_ms`, `final_beam`, `language`, ...); the worker validates them. |
|
||||
| `chatChannels` | `[]` | `{id, name, description, read (true), send (true)}` per channel. |
|
||||
| `voiceWorker.python` | `python3` (`python.exe` on Windows) | Python of the worker venv. |
|
||||
| `voiceWorker.cwd` | the repo's `src/` | Folder holding the `voice_worker` package. |
|
||||
| `voiceWorker.args` | `[]` | Extra worker arguments, e.g. `["--melo-python", "<MeloTTS venv python>"]`. |
|
||||
|
||||
## Voice worker process
|
||||
|
||||
The app starts `python -m voice_worker --host 127.0.0.1 --port <port> <args>` in `voiceWorker.cwd`.
|
||||
Each launch makes a new random token (32 bytes) and passes it only in `JARVIS_VOICE_TOKEN`. The port is
|
||||
a free loopback port chosen once at start. If the worker exits, it is started again on the same port
|
||||
with the same token after 1 s, doubling to at most 30 s while it keeps failing (reset once it prints
|
||||
`READY`). Stopping sends SIGTERM and kills it after 5 s.
|
||||
|
||||
## Worker link
|
||||
|
||||
One WebSocket to the worker, `hello` with the token, the Discord PCM format both ways
|
||||
(48 kHz stereo), and the STT settings (`voice.stt` with `hotwords` = `voice.wakeWords`). If the
|
||||
connection drops, the client reconnects with backoff (0.5 s doubling to 5 s) and says hello again with
|
||||
the same settings, including changes made since with `configure()`. While disconnected, audio frames
|
||||
are dropped (not queued), and every TTS request in flight ends as failed, so playback moves on. A TTS
|
||||
request the worker refuses (`tts_unavailable`) also ends as failed. TTS audio for request ids the client
|
||||
did not open is ignored.
|
||||
|
||||
## Voice
|
||||
|
||||
One voice session at a time, in the configured server. Joining another channel leaves the current one.
|
||||
|
||||
- **Join** (slash command `join`, auto-join, or `join(channelId)`): connect with DAVE end-to-end
|
||||
encryption (the library's default), not deafened, then post `voice.joinNotice` in the voice
|
||||
channel's chat. A notice that cannot be posted (no permission) is logged and the bot stays.
|
||||
- **Hearing**: Discord's speaking start for a user starts one long-lived Opus subscription for them,
|
||||
decoded with opusscript to 48 kHz stereo PCM and sent to the worker in 20 ms frames under the user's
|
||||
ID. The bot's own audio is never heard; other bots are skipped unless `voice.listenToBots`.
|
||||
- At most `voice.maxSpeakers` people are transcribed at once. Someone who starts while the limit is
|
||||
reached is not heard until their next speaking start after a place frees up.
|
||||
- Discord's speaking end (no packets for 100 ms) sends `speaking_end`.
|
||||
- A subscription that fails (a packet that cannot be decrypted closes it) is opened again at once
|
||||
while the person is still being heard.
|
||||
- When a person leaves the channel, the bot sends `speaker_leave`, so their utterance finishes and
|
||||
the worker frees their place.
|
||||
- **Events** for the layers above: `speech_start`, `partial` and `final` with the speaker's ID and
|
||||
server display name; `speak_start` and `speak_end`; `joined` and `left` (with a reason: `command`,
|
||||
`moved`, `disconnected`, `connect_failed`, `shutdown`).
|
||||
- **Speaking**: `say(text, more?)` queues a TTS request and returns its id (`null` when not in a voice
|
||||
channel); `continueText(id, text, more?)` adds text to an open request. Each sentence the worker
|
||||
sends becomes one clip; clips play in sentence order as soon as they arrive, and requests play first
|
||||
come, first served. A sentence that failed is skipped and the rest still play. `speak_end` reports
|
||||
how many sentences played and whether the request was cancelled or failed.
|
||||
- **Stop** (slash command `stop`, or `stop()`): the current clip stops at once (no padding frames),
|
||||
every queued request is dropped, and unfinished requests are cancelled at the worker; each ends with
|
||||
`cancelled: true`. Stopping by voice ("자비스, 멈춰") needs call detection and comes in M4.
|
||||
- **Leave** (slash command `leave`, or `leave()`): stop speaking, close every subscription (sending
|
||||
`speaker_leave`), and disconnect. A connection lost for more than 5 s (kicked, channel deleted) also
|
||||
ends the session. When someone moves the bot to another channel, the session follows it.
|
||||
|
||||
## Slash command
|
||||
|
||||
One guild command, `/<commandName>` with subcommands `join` (the caller's voice channel), `leave`
|
||||
and `stop`, registered on every start with a create-or-update by name, so the app's other commands in
|
||||
the server are left alone. A failed registration (for example a bot invited without the
|
||||
`applications.commands` scope) is logged and the bot runs on without it. Usable only in the configured
|
||||
server. Replies are in Korean, ephemeral by
|
||||
default. Renaming `commandName` leaves the old command registered until it is removed (M5 setup).
|
||||
|
||||
## Chat channels
|
||||
|
||||
- The bot asks for the message intents only when some registered channel has reading on. Reading
|
||||
needs the Message Content intent, which is checked in the application's flags before login. When it
|
||||
is off, the bot still logs in, voice works, and an error explains how to turn it on; chat reads then
|
||||
fail with a clear error instead of returning empty messages.
|
||||
- **Live**: every new message in a readable registered channel, except the bot's own, is emitted as a
|
||||
`chat` event: channel, message and author IDs, author display name, whether the author is a bot, the
|
||||
text, and the time.
|
||||
- **History**: `readChat(channelId, limit)` returns the latest 1-100 messages, oldest first, with
|
||||
`fromSelf` marking the bot's own.
|
||||
- **Send**: `sendChat(channelId, text)` sends to a registered channel with sending on, split at
|
||||
Discord's 2,000-character limit at the last newline or space (never inside an emoji), and returns
|
||||
the message ids.
|
||||
- A channel that is not registered, or has the needed switch off, is refused with `ChatAccessError`.
|
||||
|
||||
## Tests
|
||||
|
||||
- `npm test` (Vitest): playback order, gaps, stop and cancel; speaker limit; chat access and
|
||||
splitting; settings; the worker link against a fake server (hello, events, reconnect, failed TTS);
|
||||
the worker process with a stand-in `voice_worker` package (token only in the environment, restart on
|
||||
the same port, stop).
|
||||
- `JARVIS_VOICE_PYTHON=<venv python> npm test` also runs the worker link against the real Python
|
||||
worker server with fake engines (`test/fixtures/fake_engine_worker.py`).
|
||||
- `test/live/discord.live.test.ts` (opt-in, `JARVIS_LIVE_DISCORD=1`, see the file header): two bots in
|
||||
one voice channel with the real worker on the GPU. It refuses to run if a person is in the channel
|
||||
or either bot is already in a voice channel. It checks command registration, transcription of
|
||||
recorded sentences played into the channel, speech the other bot hears and the STT reads back, stop,
|
||||
chat events, history, sending without pings, and leaving.
|
||||
@@ -1,32 +0,0 @@
|
||||
/** Binary frame layout shared with the Python voice worker (src/voice_worker/protocol.py). */
|
||||
|
||||
export const KIND_AUDIO = 0x01;
|
||||
export const KIND_TTS = 0x02;
|
||||
|
||||
function encodeId(value: string): Buffer {
|
||||
const raw = Buffer.from(value, "utf8");
|
||||
if (raw.length === 0 || raw.length > 255) throw new RangeError("id must be 1-255 bytes of UTF-8");
|
||||
return Buffer.concat([Buffer.from([raw.length]), raw]);
|
||||
}
|
||||
|
||||
/** 0x01 | u8 n | speaker id | PCM16LE in the hello "audio" format. */
|
||||
export function encodeAudioFrame(speaker: string, pcm: Buffer): Buffer {
|
||||
return Buffer.concat([Buffer.from([KIND_AUDIO]), encodeId(speaker), pcm]);
|
||||
}
|
||||
|
||||
export interface TtsFrame {
|
||||
id: string;
|
||||
seq: number;
|
||||
pcm: Buffer;
|
||||
}
|
||||
|
||||
/** 0x02 | u8 n | request id | u32 LE seq | PCM16LE in the hello "tts" format. Null if malformed. */
|
||||
export function decodeTtsFrame(frame: Buffer): TtsFrame | null {
|
||||
if (frame.length < 2 || frame[0] !== KIND_TTS) return null;
|
||||
const n = frame[1]!;
|
||||
const end = 2 + n;
|
||||
if (n === 0 || frame.length < end + 4) return null;
|
||||
const id = frame.subarray(2, end).toString("utf8");
|
||||
const seq = frame.readUInt32LE(end);
|
||||
return { id, seq, pcm: frame.subarray(end + 4) };
|
||||
}
|
||||
@@ -1,25 +0,0 @@
|
||||
/**
|
||||
* Debug logging for the app, the TypeScript side of `src/jarvis/debug.py`.
|
||||
*
|
||||
* On only when JARVIS_DEBUG=1. Messages carry ids, sizes, counts and timings, never what anyone said
|
||||
* or wrote (no transcripts, no chat text, no reply text).
|
||||
*/
|
||||
export function debugLog(message: string, category = "app"): void {
|
||||
if (process.env.JARVIS_DEBUG !== "1") return;
|
||||
try {
|
||||
const pad = Math.max(0, 10 - category.length);
|
||||
const left = Math.floor(pad / 2);
|
||||
process.stderr.write(`[${" ".repeat(left)}${category}${" ".repeat(pad - left)}] ${message}\n`);
|
||||
} catch {
|
||||
/* logging must never break the app */
|
||||
}
|
||||
}
|
||||
|
||||
/** Errors are always printed; the same rule applies: no transcript or chat text in the message. */
|
||||
export function errorLog(message: string, category = "app"): void {
|
||||
try {
|
||||
process.stderr.write(`[${category}] ERROR ${message}\n`);
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
@@ -1,138 +0,0 @@
|
||||
/**
|
||||
* The bot's mouth: turns text into queued speech, one request after another.
|
||||
*
|
||||
* Each request goes to the voice worker as TTS; the worker sends back one PCM chunk per sentence.
|
||||
* Chunks play in order as soon as they arrive, so the first sentence plays while later ones are still
|
||||
* being made. Requests play first come, first served. stop() silences everything at once and cancels
|
||||
* what the worker has not made yet.
|
||||
*/
|
||||
import { EventEmitter } from "node:events";
|
||||
import { debugLog } from "./log.js";
|
||||
import type { TtsDone } from "./workerClient.js";
|
||||
|
||||
export interface TtsBackend {
|
||||
tts(id: string, text: string, more?: boolean): boolean;
|
||||
ttsCancel(id: string): void;
|
||||
}
|
||||
|
||||
/** Plays one PCM clip at a time; the owner calls Playback.clipEnded() when a clip finishes. */
|
||||
export interface ClipSink {
|
||||
play(pcm: Buffer): void;
|
||||
stop(): void;
|
||||
}
|
||||
|
||||
export interface PlaybackEnd {
|
||||
id: string;
|
||||
/** Sentences that were played. */
|
||||
played: number;
|
||||
cancelled: boolean;
|
||||
failed: boolean;
|
||||
}
|
||||
|
||||
interface Request {
|
||||
id: string;
|
||||
chunks: Map<number, Buffer>;
|
||||
next: number;
|
||||
done: boolean;
|
||||
failed: boolean;
|
||||
played: number;
|
||||
started: boolean;
|
||||
}
|
||||
|
||||
export class Playback extends EventEmitter<{ start: [string]; end: [PlaybackEnd] }> {
|
||||
private queue: Request[] = [];
|
||||
private byId = new Map<string, Request>();
|
||||
private playing = false;
|
||||
private counter = 0;
|
||||
|
||||
constructor(private readonly backend: TtsBackend, private readonly sink: ClipSink) {
|
||||
super();
|
||||
}
|
||||
|
||||
/** True while a clip is playing or a request is waiting for its audio. */
|
||||
get busy(): boolean {
|
||||
return this.playing || this.queue.length > 0;
|
||||
}
|
||||
|
||||
/** Queue `text` as a new request. With `more`, add text later with continueText(). Returns its id. */
|
||||
say(text: string, more = false): string {
|
||||
const id = `r${++this.counter}`;
|
||||
const req: Request = { id, chunks: new Map(), next: 0, done: false, failed: false, played: 0, started: false };
|
||||
this.queue.push(req);
|
||||
this.byId.set(id, req);
|
||||
debugLog(`say id=${id} chars=${text.length} more=${more} queued=${this.queue.length}`, "play");
|
||||
this.backend.tts(id, text, more);
|
||||
return id;
|
||||
}
|
||||
|
||||
/** More text for an open request (an LLM answer that is still streaming). */
|
||||
continueText(id: string, text: string, more = false): void {
|
||||
if (this.byId.has(id)) this.backend.tts(id, text, more);
|
||||
}
|
||||
|
||||
/** A sentence of audio arrived from the worker. */
|
||||
audio(id: string, seq: number, pcm: Buffer): void {
|
||||
const req = this.byId.get(id);
|
||||
if (!req) return;
|
||||
req.chunks.set(seq, pcm);
|
||||
this.pump();
|
||||
}
|
||||
|
||||
/** The worker finished a request (all sentences sent, cancelled, or failed). */
|
||||
done(d: TtsDone): void {
|
||||
const req = this.byId.get(d.id);
|
||||
if (!req) return;
|
||||
req.done = true;
|
||||
req.failed = d.failed;
|
||||
this.pump();
|
||||
}
|
||||
|
||||
/** The sink finished playing the current clip. */
|
||||
clipEnded(): void {
|
||||
this.playing = false;
|
||||
this.pump();
|
||||
}
|
||||
|
||||
/** Silence now: stop the clip, drop every queued request, cancel the worker's work. */
|
||||
stop(): number {
|
||||
const stopped = this.queue.splice(0);
|
||||
this.byId.clear();
|
||||
const wasPlaying = this.playing;
|
||||
this.playing = false;
|
||||
for (const req of stopped) if (!req.done) this.backend.ttsCancel(req.id);
|
||||
if (wasPlaying) this.sink.stop();
|
||||
for (const req of stopped) this.emit("end", { id: req.id, played: req.played, cancelled: true, failed: false });
|
||||
debugLog(`stop requests=${stopped.length} was_playing=${wasPlaying}`, "play");
|
||||
return stopped.length;
|
||||
}
|
||||
|
||||
private pump(): void {
|
||||
while (!this.playing) {
|
||||
const head = this.queue[0];
|
||||
if (!head) return;
|
||||
let pcm = head.chunks.get(head.next);
|
||||
if (!pcm && head.done && head.chunks.size) {
|
||||
// A sentence failed: skip its gap and play what did arrive.
|
||||
head.next = Math.min(...head.chunks.keys());
|
||||
pcm = head.chunks.get(head.next);
|
||||
}
|
||||
if (pcm) {
|
||||
head.chunks.delete(head.next);
|
||||
head.next++;
|
||||
head.played++;
|
||||
if (!head.started) {
|
||||
head.started = true;
|
||||
this.emit("start", head.id);
|
||||
}
|
||||
this.playing = true;
|
||||
this.sink.play(pcm);
|
||||
return;
|
||||
}
|
||||
if (!head.done) return;
|
||||
this.queue.shift();
|
||||
this.byId.delete(head.id);
|
||||
debugLog(`said id=${head.id} sentences=${head.played} failed=${head.failed}`, "play");
|
||||
this.emit("end", { id: head.id, played: head.played, cancelled: false, failed: head.failed });
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,179 +0,0 @@
|
||||
/**
|
||||
* App settings: `<data folder>/config/settings.json` (see app/src/core/discord_core.spec.md).
|
||||
*
|
||||
* Every key is optional in the file; missing keys take the defaults below. A bad value stops the app
|
||||
* with a message naming the key, rather than running with a setting the user did not choose.
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
export interface ChatChannelSettings {
|
||||
id: string;
|
||||
/** What the channel is for, as the user describes it; given to the assistant (M3). */
|
||||
name: string;
|
||||
description: string;
|
||||
/** The assistant may read this channel's messages. */
|
||||
read: boolean;
|
||||
/** The assistant may send messages to this channel. */
|
||||
send: boolean;
|
||||
}
|
||||
|
||||
export interface Settings {
|
||||
discord: {
|
||||
/** The one server the bot works in. Slash commands are registered there. */
|
||||
guildId: string | null;
|
||||
/** Voice channel to join on start ("자동 입장"); null waits for the join command. */
|
||||
voiceChannelId: string | null;
|
||||
commandName: string;
|
||||
/** Slash command replies only the caller sees. */
|
||||
ephemeralReplies: boolean;
|
||||
};
|
||||
voice: {
|
||||
/** Most people transcribed at once (1-10). Others are not heard until someone stops. */
|
||||
maxSpeakers: number;
|
||||
/** Transcribe other bots too (music bots are ignored by default). */
|
||||
listenToBots: boolean;
|
||||
/** Posted in the voice channel's chat on every join (privacy.md). */
|
||||
joinNotice: string;
|
||||
/** Wake words; passed to the STT as a hint now, used for call detection from M4. */
|
||||
wakeWords: string[];
|
||||
/** Voice worker STT settings (voice_worker.spec.md); the worker validates them. */
|
||||
stt: Record<string, unknown>;
|
||||
};
|
||||
chatChannels: ChatChannelSettings[];
|
||||
voiceWorker: {
|
||||
/** Python of the voice worker venv. */
|
||||
python: string;
|
||||
/** Directory holding the `voice_worker` package (the repo's `src/`). */
|
||||
cwd: string;
|
||||
/** Extra worker arguments, e.g. ["--melo-python", "<MeloTTS venv python>"]. */
|
||||
args: string[];
|
||||
};
|
||||
}
|
||||
|
||||
export class SettingsError extends Error {}
|
||||
|
||||
export const DEFAULT_JOIN_NOTICE =
|
||||
"자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다.";
|
||||
|
||||
const SNOWFLAKE = /^\d{17,20}$/;
|
||||
// Discord's rule for command names.
|
||||
const COMMAND_NAME = /^[-_\p{L}\p{N}\p{sc=Deva}\p{sc=Thai}]{1,32}$/u;
|
||||
|
||||
export function defaultSettings(): Settings {
|
||||
return {
|
||||
discord: { guildId: null, voiceChannelId: null, commandName: "자비스", ephemeralReplies: true },
|
||||
voice: { maxSpeakers: 10, listenToBots: false, joinNotice: DEFAULT_JOIN_NOTICE, wakeWords: ["자비스"], stt: {} },
|
||||
chatChannels: [],
|
||||
voiceWorker: {
|
||||
python: process.platform === "win32" ? "python.exe" : "python3",
|
||||
cwd: fileURLToPath(new URL("../../../src", import.meta.url)),
|
||||
args: [],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
type Obj = Record<string, unknown>;
|
||||
|
||||
function isObj(v: unknown): v is Obj {
|
||||
return typeof v === "object" && v !== null && !Array.isArray(v);
|
||||
}
|
||||
|
||||
function section(raw: Obj, key: string): Obj {
|
||||
const v = raw[key];
|
||||
if (v === undefined) return {};
|
||||
if (!isObj(v)) throw new SettingsError(`${key} must be an object`);
|
||||
return v;
|
||||
}
|
||||
|
||||
function pick<T>(obj: Obj, key: string, path: string, fallback: T, check: (v: unknown) => v is T, what: string): T {
|
||||
const v = obj[key];
|
||||
if (v === undefined) return fallback;
|
||||
if (!check(v)) throw new SettingsError(`${path}.${key} must be ${what}`);
|
||||
return v;
|
||||
}
|
||||
|
||||
const isStr = (v: unknown): v is string => typeof v === "string";
|
||||
const isBool = (v: unknown): v is boolean => typeof v === "boolean";
|
||||
const isSnowflakeOrNull = (v: unknown): v is string | null => v === null || (isStr(v) && SNOWFLAKE.test(v));
|
||||
const isStrList = (v: unknown): v is string[] => Array.isArray(v) && v.every(isStr);
|
||||
|
||||
/** Validate a parsed settings object over the defaults. */
|
||||
export function parseSettings(raw: unknown, defaults: Settings = defaultSettings()): Settings {
|
||||
if (!isObj(raw)) throw new SettingsError("settings must be a JSON object");
|
||||
const d = section(raw, "discord");
|
||||
const v = section(raw, "voice");
|
||||
const w = section(raw, "voiceWorker");
|
||||
|
||||
const commandName = pick(d, "commandName", "discord", defaults.discord.commandName, isStr, "a string");
|
||||
if (!COMMAND_NAME.test(commandName) || commandName !== commandName.toLowerCase()) {
|
||||
throw new SettingsError("discord.commandName must be 1-32 lowercase letters, digits, - or _");
|
||||
}
|
||||
const maxSpeakers = pick(v, "maxSpeakers", "voice", defaults.voice.maxSpeakers,
|
||||
(x): x is number => Number.isInteger(x) && (x as number) >= 1 && (x as number) <= 10, "an integer 1-10");
|
||||
const wakeWords = pick(v, "wakeWords", "voice", defaults.voice.wakeWords, isStrList, "a list of strings")
|
||||
.map((s) => s.trim()).filter(Boolean);
|
||||
const joinNotice = pick(v, "joinNotice", "voice", defaults.voice.joinNotice, isStr, "a string").trim();
|
||||
if (!joinNotice) throw new SettingsError("voice.joinNotice must not be empty: people must be told they are transcribed");
|
||||
if (joinNotice.length > 2000) throw new SettingsError("voice.joinNotice must be at most 2000 characters");
|
||||
|
||||
const chatRaw = raw.chatChannels ?? [];
|
||||
if (!Array.isArray(chatRaw)) throw new SettingsError("chatChannels must be a list");
|
||||
const seen = new Set<string>();
|
||||
const chatChannels = chatRaw.map((c, i): ChatChannelSettings => {
|
||||
const path = `chatChannels[${i}]`;
|
||||
if (!isObj(c)) throw new SettingsError(`${path} must be an object`);
|
||||
const id = c.id;
|
||||
if (!isStr(id) || !SNOWFLAKE.test(id)) throw new SettingsError(`${path}.id must be a Discord channel ID`);
|
||||
if (seen.has(id)) throw new SettingsError(`${path}.id is listed twice`);
|
||||
seen.add(id);
|
||||
return {
|
||||
id,
|
||||
name: pick(c, "name", path, "", isStr, "a string"),
|
||||
description: pick(c, "description", path, "", isStr, "a string"),
|
||||
read: pick(c, "read", path, true, isBool, "true or false"),
|
||||
send: pick(c, "send", path, true, isBool, "true or false"),
|
||||
};
|
||||
});
|
||||
|
||||
return {
|
||||
discord: {
|
||||
guildId: pick(d, "guildId", "discord", defaults.discord.guildId, isSnowflakeOrNull, "a Discord server ID or null"),
|
||||
voiceChannelId: pick(d, "voiceChannelId", "discord", defaults.discord.voiceChannelId, isSnowflakeOrNull,
|
||||
"a Discord channel ID or null"),
|
||||
commandName,
|
||||
ephemeralReplies: pick(d, "ephemeralReplies", "discord", defaults.discord.ephemeralReplies, isBool, "true or false"),
|
||||
},
|
||||
voice: {
|
||||
maxSpeakers,
|
||||
listenToBots: pick(v, "listenToBots", "voice", defaults.voice.listenToBots, isBool, "true or false"),
|
||||
joinNotice,
|
||||
wakeWords,
|
||||
stt: pick(v, "stt", "voice", defaults.voice.stt, isObj, "an object"),
|
||||
},
|
||||
chatChannels,
|
||||
voiceWorker: {
|
||||
python: pick(w, "python", "voiceWorker", defaults.voiceWorker.python, isStr, "a string"),
|
||||
cwd: pick(w, "cwd", "voiceWorker", defaults.voiceWorker.cwd, isStr, "a string"),
|
||||
args: pick(w, "args", "voiceWorker", defaults.voiceWorker.args, isStrList, "a list of strings"),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Read and validate the settings file. A missing file means all defaults. */
|
||||
export function loadSettings(path: string): Settings {
|
||||
let text: string;
|
||||
try {
|
||||
text = readFileSync(path, "utf8");
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException).code === "ENOENT") return defaultSettings();
|
||||
throw e;
|
||||
}
|
||||
let raw: unknown;
|
||||
try {
|
||||
raw = JSON.parse(text.replace(/^/, ""));
|
||||
} catch (e) {
|
||||
throw new SettingsError(`${path} is not valid JSON: ${(e as Error).message}`);
|
||||
}
|
||||
return parseSettings(raw);
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
/**
|
||||
* Who is being transcribed right now. At most `max` people at once (settings voice.maxSpeakers);
|
||||
* someone who starts while the limit is reached is not heard until their next start after the
|
||||
* others free a place.
|
||||
*/
|
||||
export class SpeakerGate {
|
||||
private active = new Set<string>();
|
||||
|
||||
constructor(private max: number) {}
|
||||
|
||||
/** Discord says `id` started speaking. True if their audio goes to the STT. */
|
||||
start(id: string): boolean {
|
||||
if (this.active.has(id)) return true;
|
||||
if (this.active.size >= this.max) return false;
|
||||
this.active.add(id);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Discord says `id` stopped (or left). True if they were being transcribed. */
|
||||
end(id: string): boolean {
|
||||
return this.active.delete(id);
|
||||
}
|
||||
|
||||
isActive(id: string): boolean {
|
||||
return this.active.has(id);
|
||||
}
|
||||
|
||||
get count(): number {
|
||||
return this.active.size;
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
this.active.clear();
|
||||
}
|
||||
}
|
||||
@@ -1,278 +0,0 @@
|
||||
/**
|
||||
* WebSocket client for the Python voice worker (src/voice_worker/voice_worker.spec.md).
|
||||
*
|
||||
* Keeps one connection open, says hello with the app's token, and reconnects with backoff when the
|
||||
* worker restarts. While disconnected, audio is dropped (the worker would not know the speaker's
|
||||
* state anyway) and every open TTS request ends as failed so playback never waits forever.
|
||||
*/
|
||||
import { EventEmitter } from "node:events";
|
||||
import WebSocket from "ws";
|
||||
import { decodeTtsFrame, encodeAudioFrame, KIND_TTS } from "./frames.js";
|
||||
import { debugLog, errorLog } from "./log.js";
|
||||
|
||||
export interface PcmFormat {
|
||||
rate: number;
|
||||
channels: 1 | 2;
|
||||
}
|
||||
|
||||
export interface WorkerClientOptions {
|
||||
url: string;
|
||||
token: string;
|
||||
audio: PcmFormat;
|
||||
tts: PcmFormat;
|
||||
stt?: Record<string, unknown>;
|
||||
/** First reconnect delay; doubles up to maxBackoffMs. */
|
||||
backoffMs?: number;
|
||||
maxBackoffMs?: number;
|
||||
helloTimeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface HelloOk {
|
||||
stt_ready: boolean;
|
||||
tts_ready: boolean;
|
||||
stt: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface SpeechStart {
|
||||
speaker: string;
|
||||
utt: number;
|
||||
}
|
||||
export interface Partial {
|
||||
speaker: string;
|
||||
utt: number;
|
||||
text: string;
|
||||
}
|
||||
export interface Final {
|
||||
speaker: string;
|
||||
utt: number;
|
||||
text: string;
|
||||
end_reason: string;
|
||||
audio_ms: number;
|
||||
wait_ms: number;
|
||||
stt_ms: number;
|
||||
}
|
||||
export interface TtsDone {
|
||||
id: string;
|
||||
chunks: number;
|
||||
cancelled: boolean;
|
||||
failed: boolean;
|
||||
}
|
||||
export interface WorkerError {
|
||||
code: string;
|
||||
message: string;
|
||||
id?: string;
|
||||
}
|
||||
|
||||
export interface WorkerClientEvents {
|
||||
ready: [HelloOk];
|
||||
disconnected: [];
|
||||
speech_start: [SpeechStart];
|
||||
partial: [Partial];
|
||||
final: [Final];
|
||||
tts_audio: [{ id: string; seq: number; pcm: Buffer }];
|
||||
tts_done: [TtsDone];
|
||||
worker_error: [WorkerError];
|
||||
}
|
||||
|
||||
export class VoiceWorkerClient extends EventEmitter<WorkerClientEvents> {
|
||||
private ws: WebSocket | null = null;
|
||||
private ready = false;
|
||||
private closed = false;
|
||||
private retryTimer: NodeJS.Timeout | null = null;
|
||||
private backoff: number;
|
||||
private stt: Record<string, unknown>;
|
||||
/** TTS requests sent and not yet done, so a lost connection can end them. */
|
||||
private openTts = new Set<string>();
|
||||
private dropped = 0;
|
||||
private hello: HelloOk | null = null;
|
||||
|
||||
constructor(private readonly opts: WorkerClientOptions) {
|
||||
super();
|
||||
this.backoff = opts.backoffMs ?? 500;
|
||||
this.stt = { ...(opts.stt ?? {}) };
|
||||
}
|
||||
|
||||
get isReady(): boolean {
|
||||
return this.ready;
|
||||
}
|
||||
|
||||
get helloOk(): HelloOk | null {
|
||||
return this.ready ? this.hello : null;
|
||||
}
|
||||
|
||||
/** Start connecting; reconnects until close(). */
|
||||
start(): void {
|
||||
this.closed = false;
|
||||
this.connect();
|
||||
}
|
||||
|
||||
/** Resolve once the worker has said hello_ok (or reject after timeoutMs). */
|
||||
waitReady(timeoutMs: number): Promise<HelloOk> {
|
||||
if (this.ready && this.hello) return Promise.resolve(this.hello);
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => {
|
||||
this.off("ready", onReady);
|
||||
reject(new Error(`voice worker not ready after ${timeoutMs} ms`));
|
||||
}, timeoutMs);
|
||||
const onReady = (h: HelloOk) => {
|
||||
clearTimeout(t);
|
||||
resolve(h);
|
||||
};
|
||||
this.once("ready", onReady);
|
||||
});
|
||||
}
|
||||
|
||||
close(): void {
|
||||
this.closed = true;
|
||||
if (this.retryTimer) clearTimeout(this.retryTimer);
|
||||
this.retryTimer = null;
|
||||
this.ws?.close(1000);
|
||||
this.ws = null;
|
||||
this.markDisconnected();
|
||||
}
|
||||
|
||||
private connect(): void {
|
||||
if (this.closed) return;
|
||||
const ws = new WebSocket(this.opts.url, { maxPayload: 2 ** 24, perMessageDeflate: false });
|
||||
this.ws = ws;
|
||||
let helloTimer: NodeJS.Timeout | null = null;
|
||||
ws.on("open", () => {
|
||||
ws.send(JSON.stringify({ type: "hello", token: this.opts.token, audio: this.opts.audio, tts: this.opts.tts,
|
||||
stt: this.stt }));
|
||||
helloTimer = setTimeout(() => ws.terminate(), this.opts.helloTimeoutMs ?? 5000);
|
||||
});
|
||||
ws.on("message", (data, isBinary) => {
|
||||
if (isBinary) return this.onBinary(data as Buffer);
|
||||
let msg: Record<string, unknown>;
|
||||
try {
|
||||
msg = JSON.parse(String(data));
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
if (msg.type === "hello_ok") {
|
||||
if (helloTimer) clearTimeout(helloTimer);
|
||||
this.ready = true;
|
||||
this.backoff = this.opts.backoffMs ?? 500;
|
||||
this.hello = msg as unknown as HelloOk;
|
||||
debugLog(`voice worker ready stt=${this.hello.stt_ready} tts=${this.hello.tts_ready}`, "worker");
|
||||
this.emit("ready", this.hello);
|
||||
return;
|
||||
}
|
||||
this.onMessage(msg);
|
||||
});
|
||||
ws.on("close", (code, reason) => {
|
||||
if (helloTimer) clearTimeout(helloTimer);
|
||||
if (this.ws !== ws) return;
|
||||
this.ws = null;
|
||||
if (code === 4400 || code === 4401) {
|
||||
errorLog(`voice worker refused the connection (${code} ${String(reason)})`, "worker");
|
||||
}
|
||||
this.markDisconnected();
|
||||
this.scheduleReconnect();
|
||||
});
|
||||
ws.on("error", (e) => {
|
||||
// "close" follows; connection refused is normal while the worker is still loading.
|
||||
debugLog(`voice worker socket error: ${(e as NodeJS.ErrnoException).code ?? e.message}`, "worker");
|
||||
});
|
||||
}
|
||||
|
||||
private markDisconnected(): void {
|
||||
const was = this.ready;
|
||||
this.ready = false;
|
||||
for (const id of this.openTts) this.emit("tts_done", { id, chunks: 0, cancelled: false, failed: true });
|
||||
this.openTts.clear();
|
||||
if (was) {
|
||||
debugLog(`voice worker disconnected (dropped audio frames so far: ${this.dropped})`, "worker");
|
||||
this.emit("disconnected");
|
||||
}
|
||||
}
|
||||
|
||||
private scheduleReconnect(): void {
|
||||
if (this.closed || this.retryTimer) return;
|
||||
const delay = this.backoff;
|
||||
this.backoff = Math.min(this.backoff * 2, this.opts.maxBackoffMs ?? 5000);
|
||||
this.retryTimer = setTimeout(() => {
|
||||
this.retryTimer = null;
|
||||
this.connect();
|
||||
}, delay);
|
||||
}
|
||||
|
||||
private onBinary(buf: Buffer): void {
|
||||
if (buf[0] !== KIND_TTS) return;
|
||||
const frame = decodeTtsFrame(buf);
|
||||
if (frame && this.openTts.has(frame.id)) this.emit("tts_audio", frame);
|
||||
}
|
||||
|
||||
private onMessage(raw: Record<string, unknown>): void {
|
||||
const { type, ...msg } = raw;
|
||||
switch (type) {
|
||||
case "speech_start":
|
||||
this.emit("speech_start", msg as unknown as SpeechStart);
|
||||
break;
|
||||
case "partial":
|
||||
this.emit("partial", msg as unknown as Partial);
|
||||
break;
|
||||
case "final":
|
||||
this.emit("final", msg as unknown as Final);
|
||||
break;
|
||||
case "tts_done": {
|
||||
const done = msg as unknown as TtsDone;
|
||||
if (this.openTts.delete(done.id)) this.emit("tts_done", done);
|
||||
break;
|
||||
}
|
||||
case "error": {
|
||||
const err = msg as unknown as WorkerError;
|
||||
debugLog(`voice worker error ${err.code}`, "worker");
|
||||
this.emit("worker_error", err);
|
||||
// A request the worker refused outright never gets a tts_done.
|
||||
if (err.code === "tts_unavailable" && err.id && this.openTts.delete(err.id)) {
|
||||
this.emit("tts_done", { id: err.id, chunks: 0, cancelled: false, failed: true });
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private send(data: string | Buffer): boolean {
|
||||
if (!this.ready || !this.ws || this.ws.readyState !== WebSocket.OPEN) return false;
|
||||
this.ws.send(data);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** One frame of a speaker's PCM in the hello "audio" format. Dropped while disconnected. */
|
||||
sendAudio(speaker: string, pcm: Buffer): void {
|
||||
if (!this.send(encodeAudioFrame(speaker, pcm))) this.dropped++;
|
||||
}
|
||||
|
||||
speakingEnd(speaker: string): void {
|
||||
this.send(JSON.stringify({ type: "speaking_end", speaker }));
|
||||
}
|
||||
|
||||
speakerLeave(speaker: string): void {
|
||||
this.send(JSON.stringify({ type: "speaker_leave", speaker }));
|
||||
}
|
||||
|
||||
/** Change STT settings now and for every later connection. */
|
||||
configure(changes: Record<string, unknown>): void {
|
||||
Object.assign(this.stt, changes);
|
||||
this.send(JSON.stringify({ type: "config", ...changes }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Speak `text` as request `id`; with `more` the request stays open for more text.
|
||||
* Returns false (and the request is already failed) when the worker is not connected.
|
||||
*/
|
||||
tts(id: string, text: string, more = false): boolean {
|
||||
const sent = this.send(JSON.stringify({ type: "tts", id, text, more }));
|
||||
if (sent) {
|
||||
this.openTts.add(id);
|
||||
} else if (!this.openTts.has(id)) {
|
||||
queueMicrotask(() => this.emit("tts_done", { id, chunks: 0, cancelled: false, failed: true }));
|
||||
}
|
||||
return sent;
|
||||
}
|
||||
|
||||
ttsCancel(id: string): void {
|
||||
this.send(JSON.stringify({ type: "tts_cancel", id }));
|
||||
}
|
||||
}
|
||||
@@ -1,119 +0,0 @@
|
||||
/**
|
||||
* Starts the Python voice worker as a child process and restarts it if it dies.
|
||||
*
|
||||
* Each app launch makes a new random token and hands it over in the environment
|
||||
* (JARVIS_VOICE_TOKEN), never on the command line. The worker listens on 127.0.0.1 only.
|
||||
*/
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { EventEmitter } from "node:events";
|
||||
import { createServer } from "node:net";
|
||||
import { createInterface } from "node:readline";
|
||||
import { debugLog, errorLog } from "./log.js";
|
||||
|
||||
export interface WorkerProcessOptions {
|
||||
python: string;
|
||||
cwd: string;
|
||||
args: string[];
|
||||
/** Fixed port; 0 picks a free one once at start (kept across restarts). */
|
||||
port?: number;
|
||||
/** Extra environment for the worker (merged over the app's). */
|
||||
env?: Record<string, string>;
|
||||
restartDelayMs?: number;
|
||||
maxRestartDelayMs?: number;
|
||||
}
|
||||
|
||||
export async function freePort(): Promise<number> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const srv = createServer();
|
||||
srv.once("error", reject);
|
||||
srv.listen(0, "127.0.0.1", () => {
|
||||
const addr = srv.address();
|
||||
srv.close(() => (typeof addr === "object" && addr ? resolve(addr.port) : reject(new Error("no port"))));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export interface WorkerProcessEvents {
|
||||
/** The worker printed READY: models are loaded and it accepts connections. */
|
||||
ready: [];
|
||||
exit: [{ code: number | null; signal: NodeJS.Signals | null }];
|
||||
}
|
||||
|
||||
export class VoiceWorkerProcess extends EventEmitter<WorkerProcessEvents> {
|
||||
readonly token = randomBytes(32).toString("base64url");
|
||||
private child: ChildProcess | null = null;
|
||||
private stopping = false;
|
||||
private restartTimer: NodeJS.Timeout | null = null;
|
||||
private delay: number;
|
||||
private port = 0;
|
||||
private startedAt = 0;
|
||||
restarts = 0;
|
||||
|
||||
constructor(private readonly opts: WorkerProcessOptions) {
|
||||
super();
|
||||
this.delay = opts.restartDelayMs ?? 1000;
|
||||
}
|
||||
|
||||
get url(): string {
|
||||
return `ws://127.0.0.1:${this.port}`;
|
||||
}
|
||||
|
||||
async start(): Promise<void> {
|
||||
this.stopping = false;
|
||||
this.port = this.opts.port || (await freePort());
|
||||
this.spawnChild();
|
||||
}
|
||||
|
||||
private spawnChild(): void {
|
||||
const args = ["-m", "voice_worker", "--host", "127.0.0.1", "--port", String(this.port), ...this.opts.args];
|
||||
this.startedAt = Date.now();
|
||||
const child = spawn(this.opts.python, args, {
|
||||
cwd: this.opts.cwd,
|
||||
env: { ...process.env, ...this.opts.env, JARVIS_VOICE_TOKEN: this.token, PYTHONUNBUFFERED: "1" },
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
windowsHide: true,
|
||||
});
|
||||
this.child = child;
|
||||
debugLog(`voice worker started pid=${child.pid} port=${this.port}`, "worker");
|
||||
// The worker's own logs hold timings and counts only (voice_worker.spec.md), so they can be passed on.
|
||||
createInterface({ input: child.stdout! }).on("line", (line) => {
|
||||
if (line.startsWith("READY ")) {
|
||||
debugLog(`voice worker READY after ${Date.now() - this.startedAt} ms`, "worker");
|
||||
this.delay = this.opts.restartDelayMs ?? 1000;
|
||||
this.emit("ready");
|
||||
} else debugLog(line, "worker");
|
||||
});
|
||||
createInterface({ input: child.stderr! }).on("line", (line) => debugLog(line, "worker"));
|
||||
child.once("error", (e) => errorLog(`voice worker could not start: ${e.message}`, "worker"));
|
||||
child.once("close", (code, signal) => {
|
||||
if (this.child === child) this.child = null;
|
||||
this.emit("exit", { code, signal });
|
||||
if (this.stopping) return;
|
||||
errorLog(`voice worker exited (code ${code}, signal ${signal}); restarting in ${this.delay} ms`, "worker");
|
||||
this.restartTimer = setTimeout(() => {
|
||||
this.restartTimer = null;
|
||||
this.restarts++;
|
||||
this.spawnChild();
|
||||
}, this.delay);
|
||||
this.delay = Math.min(this.delay * 2, this.opts.maxRestartDelayMs ?? 30_000);
|
||||
});
|
||||
}
|
||||
|
||||
/** Stop the worker and do not restart it. Resolves when it has exited. */
|
||||
async stop(graceMs = 5000): Promise<void> {
|
||||
this.stopping = true;
|
||||
if (this.restartTimer) clearTimeout(this.restartTimer);
|
||||
this.restartTimer = null;
|
||||
const child = this.child;
|
||||
if (!child || child.exitCode !== null || child.signalCode !== null) return;
|
||||
await new Promise<void>((resolve) => {
|
||||
const kill = setTimeout(() => child.kill("SIGKILL"), graceMs);
|
||||
child.once("close", () => {
|
||||
clearTimeout(kill);
|
||||
resolve();
|
||||
});
|
||||
child.kill("SIGTERM");
|
||||
});
|
||||
}
|
||||
}
|
||||
838
app/src/discord.rs
Normal file
838
app/src/discord.rs
Normal file
@@ -0,0 +1,838 @@
|
||||
//! The Discord voice core (`app/src/discord_core.spec.md`).
|
||||
//!
|
||||
//! One bot, one server, one voice channel at a time. Hears each speaker separately and streams their
|
||||
//! PCM to the voice worker, speaks text through the worker's TTS, and reads and writes the registered
|
||||
//! chat channels. It has no assistant logic: it reports what was heard and does what it is told.
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
use std::sync::{Arc, Mutex, OnceLock};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use serenity::all::{
|
||||
Cache, ChannelId, ChannelType, Client, CommandDataOptionValue, CommandInteraction, CommandOptionType, Context,
|
||||
CreateAllowedMentions, CreateCommand, CreateCommandOption, CreateInteractionResponse,
|
||||
CreateInteractionResponseMessage, CreateMessage, EditInteractionResponse, EventHandler, GatewayIntents, GetMessages,
|
||||
GuildId, Http, InstallationContext, Interaction, InteractionContext, Message, Ready, ShardManager, UserId,
|
||||
VoiceState,
|
||||
};
|
||||
use songbird::driver::DecodeMode;
|
||||
use songbird::input::Input;
|
||||
use songbird::tracks::TrackHandle;
|
||||
use songbird::{Call, Config, Event, EventContext, EventHandler as VoiceEventHandler, SerenityInit, Songbird, TrackEvent};
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
use crate::chat::{has_message_content_intent, split_message, ChatAccessError, ChatChannels, ChatMessage, MESSAGE_LIMIT};
|
||||
use crate::playback::{ClipSink, Playback, PlaybackEnd, PlaybackEvent};
|
||||
use crate::receive::{opus_payload, OpusFrames, SpeakerStream};
|
||||
use crate::settings::Settings;
|
||||
use crate::speakers::SpeakerGate;
|
||||
use crate::worker_client::{Final, WorkerClient, WorkerEvent};
|
||||
use crate::{debug_log, error_log};
|
||||
|
||||
/// No packet for this long and a speaker counts as stopped (Discord.js uses the same 100 ms).
|
||||
const SPEAKING_END: Duration = Duration::from_millis(100);
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Heard {
|
||||
pub speaker_id: u64,
|
||||
pub speaker_name: String,
|
||||
pub utt: u32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct HeardFinal {
|
||||
pub heard: Heard,
|
||||
pub text: String,
|
||||
pub end_reason: String,
|
||||
pub audio_ms: u64,
|
||||
pub wait_ms: u64,
|
||||
pub stt_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum CoreEvent {
|
||||
Ready { user_id: u64, tag: String },
|
||||
Joined { guild_id: u64, channel_id: u64 },
|
||||
Left { guild_id: u64, channel_id: u64, reason: String },
|
||||
SpeechStart(Heard),
|
||||
Partial(Heard, String),
|
||||
Final(HeardFinal),
|
||||
SpeakStart(String),
|
||||
SpeakEnd(PlaybackEnd),
|
||||
Chat(ChatMessage),
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct CoreError(pub String);
|
||||
|
||||
impl std::fmt::Display for CoreError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str(&self.0)
|
||||
}
|
||||
}
|
||||
impl std::error::Error for CoreError {}
|
||||
impl From<ChatAccessError> for CoreError {
|
||||
fn from(e: ChatAccessError) -> Self {
|
||||
CoreError(e.0)
|
||||
}
|
||||
}
|
||||
fn cerr<E: std::fmt::Display>(what: &str) -> impl FnOnce(E) -> CoreError + '_ {
|
||||
move |e| CoreError(format!("{what}: {e}"))
|
||||
}
|
||||
|
||||
// --- playback sink: one player task owns the songbird track of the current clip -------------------
|
||||
|
||||
enum PlayerCmd {
|
||||
Play(Vec<u8>),
|
||||
Stop,
|
||||
SetCall(Option<Arc<tokio::sync::Mutex<Call>>>),
|
||||
Ended(u64),
|
||||
}
|
||||
|
||||
struct PlayerSink(mpsc::UnboundedSender<PlayerCmd>);
|
||||
|
||||
impl ClipSink for PlayerSink {
|
||||
fn play(&mut self, pcm: Vec<u8>) {
|
||||
let _ = self.0.send(PlayerCmd::Play(pcm));
|
||||
}
|
||||
fn stop(&mut self) {
|
||||
let _ = self.0.send(PlayerCmd::Stop);
|
||||
}
|
||||
}
|
||||
|
||||
struct TrackEnded {
|
||||
tx: mpsc::UnboundedSender<PlayerCmd>,
|
||||
generation: u64,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl VoiceEventHandler for TrackEnded {
|
||||
async fn act(&self, _: &EventContext<'_>) -> Option<Event> {
|
||||
let _ = self.tx.send(PlayerCmd::Ended(self.generation));
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// 48 kHz stereo PCM16 as a WAV, which songbird plays without resampling.
|
||||
fn wav(pcm: &[u8]) -> Vec<u8> {
|
||||
let mut w = Vec::with_capacity(44 + pcm.len());
|
||||
let len = pcm.len() as u32;
|
||||
w.extend_from_slice(b"RIFF");
|
||||
w.extend_from_slice(&(36 + len).to_le_bytes());
|
||||
w.extend_from_slice(b"WAVEfmt ");
|
||||
w.extend_from_slice(&16u32.to_le_bytes());
|
||||
w.extend_from_slice(&1u16.to_le_bytes()); // PCM
|
||||
w.extend_from_slice(&2u16.to_le_bytes()); // stereo
|
||||
w.extend_from_slice(&48000u32.to_le_bytes());
|
||||
w.extend_from_slice(&(48000u32 * 4).to_le_bytes());
|
||||
w.extend_from_slice(&4u16.to_le_bytes());
|
||||
w.extend_from_slice(&16u16.to_le_bytes());
|
||||
w.extend_from_slice(b"data");
|
||||
w.extend_from_slice(&len.to_le_bytes());
|
||||
w.extend_from_slice(pcm);
|
||||
w
|
||||
}
|
||||
|
||||
// --- core -----------------------------------------------------------------------------------------
|
||||
|
||||
struct Session {
|
||||
guild_id: u64,
|
||||
channel_id: u64,
|
||||
rx: Arc<Mutex<RxState>>,
|
||||
/// Cleared when the session ends; stops its speaking-end timer.
|
||||
alive: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct RxState {
|
||||
ssrc_user: HashMap<u32, u64>,
|
||||
streams: HashMap<u32, SpeakerStream<OpusFrames>>,
|
||||
/// People currently sending packets, with the time of their last packet.
|
||||
talking: HashMap<u64, Instant>,
|
||||
/// Everyone whose audio reached the worker this session (they get `speaker_leave`).
|
||||
heard: HashSet<u64>,
|
||||
}
|
||||
|
||||
struct Inner {
|
||||
settings: Settings,
|
||||
token: String,
|
||||
worker: WorkerClient,
|
||||
events: mpsc::UnboundedSender<CoreEvent>,
|
||||
songbird: Arc<Songbird>,
|
||||
http: OnceLock<Arc<Http>>,
|
||||
cache: OnceLock<Arc<Cache>>,
|
||||
shards: OnceLock<Arc<ShardManager>>,
|
||||
bot_id: OnceLock<u64>,
|
||||
chat: ChatChannels,
|
||||
chat_content: AtomicBool,
|
||||
gate: Mutex<SpeakerGate>,
|
||||
playback: Mutex<Playback<WorkerClient, PlayerSink>>,
|
||||
player: mpsc::UnboundedSender<PlayerCmd>,
|
||||
session: Mutex<Option<Session>>,
|
||||
/// Serialises join and leave.
|
||||
voice_lock: tokio::sync::Mutex<()>,
|
||||
names: Mutex<HashMap<u64, String>>,
|
||||
}
|
||||
|
||||
/// Handle to the core; cheap to clone.
|
||||
#[derive(Clone)]
|
||||
pub struct DiscordCore {
|
||||
inner: Arc<Inner>,
|
||||
}
|
||||
|
||||
impl DiscordCore {
|
||||
/// The core and its event stream. `worker_events` is the worker client's event stream.
|
||||
pub fn new(settings: Settings, token: String, worker: WorkerClient,
|
||||
worker_events: mpsc::UnboundedReceiver<WorkerEvent>) -> (Self, mpsc::UnboundedReceiver<CoreEvent>) {
|
||||
let (events, rx) = mpsc::unbounded_channel();
|
||||
let (player_tx, player_rx) = mpsc::unbounded_channel();
|
||||
// songbird only decrypts; the core reorders and decodes itself (see receive.rs for why).
|
||||
let config = Config::default().decode_mode(DecodeMode::Decrypt);
|
||||
let inner = Arc::new(Inner {
|
||||
gate: Mutex::new(SpeakerGate::new(settings.voice.max_speakers)),
|
||||
chat: ChatChannels::new(&settings.chat_channels),
|
||||
playback: Mutex::new(Playback::new(worker.clone(), PlayerSink(player_tx.clone()))),
|
||||
settings,
|
||||
token,
|
||||
worker,
|
||||
events,
|
||||
songbird: Songbird::serenity_from_config(config),
|
||||
http: OnceLock::new(),
|
||||
cache: OnceLock::new(),
|
||||
shards: OnceLock::new(),
|
||||
bot_id: OnceLock::new(),
|
||||
chat_content: AtomicBool::new(false),
|
||||
player: player_tx,
|
||||
session: Mutex::new(None),
|
||||
voice_lock: tokio::sync::Mutex::new(()),
|
||||
names: Mutex::new(HashMap::new()),
|
||||
});
|
||||
tokio::spawn(player_task(Arc::downgrade(&inner), player_rx));
|
||||
tokio::spawn(worker_task(Arc::downgrade(&inner), worker_events));
|
||||
(Self { inner }, rx)
|
||||
}
|
||||
|
||||
fn http(&self) -> Result<&Arc<Http>, CoreError> {
|
||||
self.inner.http.get().ok_or_else(|| CoreError("the Discord core has not been started".into()))
|
||||
}
|
||||
|
||||
/// The application has the Message Content intent; without it chat channels cannot be read.
|
||||
pub fn chat_content(&self) -> bool {
|
||||
self.inner.chat_content.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
pub fn bot_id(&self) -> Option<u64> {
|
||||
self.inner.bot_id.get().copied()
|
||||
}
|
||||
|
||||
/// Log in, register the slash command, and join the auto-join channel if one is set.
|
||||
pub async fn start(&self) -> Result<(), CoreError> {
|
||||
let inner = &self.inner;
|
||||
let http = Http::new(&inner.token);
|
||||
let app = http.get_current_application_info().await.map_err(cerr("could not reach Discord"))?;
|
||||
let flags = app.flags.map(|f| f.bits()).unwrap_or(0);
|
||||
let content = has_message_content_intent(flags);
|
||||
inner.chat_content.store(content, Ordering::Relaxed);
|
||||
let mut intents = GatewayIntents::GUILDS | GatewayIntents::GUILD_VOICE_STATES;
|
||||
if inner.chat.any_readable() {
|
||||
intents |= GatewayIntents::GUILD_MESSAGES;
|
||||
if content {
|
||||
intents |= GatewayIntents::MESSAGE_CONTENT;
|
||||
} else {
|
||||
error_log!("discord", "Message Content Intent is off for this bot; chat channels cannot be read. \
|
||||
Turn it on in the Discord Developer Portal (Bot > Privileged Gateway Intents).");
|
||||
}
|
||||
}
|
||||
let (ready_tx, mut ready_rx) = mpsc::unbounded_channel();
|
||||
let mut client = Client::builder(&inner.token, intents)
|
||||
.event_handler(Handler { core: self.clone(), ready: ready_tx })
|
||||
.register_songbird_with(inner.songbird.clone())
|
||||
.await
|
||||
.map_err(cerr("could not create the Discord client"))?;
|
||||
let _ = inner.http.set(client.http.clone());
|
||||
let _ = inner.cache.set(client.cache.clone());
|
||||
let _ = inner.shards.set(client.shard_manager.clone());
|
||||
let (fail_tx, mut fail_rx) = mpsc::unbounded_channel();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = client.start().await {
|
||||
let _ = fail_tx.send(e.to_string());
|
||||
}
|
||||
});
|
||||
let ready: Ready = tokio::select! {
|
||||
r = ready_rx.recv() => r.ok_or_else(|| CoreError("gateway closed before ready".into()))?,
|
||||
e = fail_rx.recv() => return Err(CoreError(format!("could not log in: {}", e.unwrap_or_default()))),
|
||||
};
|
||||
let _ = inner.bot_id.set(ready.user.id.get());
|
||||
debug_log!("discord", "logged in as {} chatContent={content}", ready.user.id);
|
||||
let _ = inner.events.send(CoreEvent::Ready { user_id: ready.user.id.get(), tag: ready.user.tag() });
|
||||
if let Err(e) = self.register_command(app.id.get()).await {
|
||||
// Voice and chat still work; the command can be registered on the next start.
|
||||
error_log!("discord", "could not register the slash command: {e}");
|
||||
}
|
||||
if let Some(ch) = inner.settings.discord.voice_channel_id {
|
||||
if let Err(e) = self.join(ch).await {
|
||||
error_log!("voice", "auto-join {ch} failed: {e}");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn register_command(&self, app_id: u64) -> Result<(), CoreError> {
|
||||
let Some(guild) = self.inner.settings.discord.guild_id else {
|
||||
error_log!("discord", "discord.guildId is not set; the slash command is not registered");
|
||||
return Ok(());
|
||||
};
|
||||
let name = &self.inner.settings.discord.command_name;
|
||||
let sub = |n: &str, d: &str| CreateCommandOption::new(CommandOptionType::SubCommand, n, d);
|
||||
let cmd = CreateCommand::new(name)
|
||||
.description("자비스 음성 비서")
|
||||
.contexts(vec![InteractionContext::Guild])
|
||||
.integration_types(vec![InstallationContext::Guild])
|
||||
.add_option(sub("join", "내가 있는 음성 채널로 자비스를 부릅니다"))
|
||||
.add_option(sub("leave", "자비스를 음성 채널에서 내보냅니다"))
|
||||
.add_option(sub("stop", "자비스가 하던 말을 멈춥니다"));
|
||||
let http = self.http()?;
|
||||
http.set_application_id(app_id.into());
|
||||
// Creates or updates this one command by name and leaves the app's other commands alone.
|
||||
GuildId::new(guild).create_command(http.as_ref(), cmd).await.map_err(cerr("create command"))?;
|
||||
debug_log!("discord", "slash command /{name} registered in {guild}");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// --- voice ------------------------------------------------------------------------------------
|
||||
|
||||
pub fn voice_channel_id(&self) -> Option<u64> {
|
||||
self.inner.session.lock().unwrap().as_ref().map(|s| s.channel_id)
|
||||
}
|
||||
|
||||
/// Join a voice channel of the configured server (leaving the current one) and post the notice.
|
||||
pub async fn join(&self, channel_id: u64) -> Result<(), CoreError> {
|
||||
let _guard = self.inner.voice_lock.lock().await;
|
||||
let http = self.http()?.clone();
|
||||
let channel = ChannelId::new(channel_id).to_channel(&http).await.map_err(cerr("could not fetch the channel"))?;
|
||||
let Some(gc) = channel.guild().filter(|c| matches!(c.kind, ChannelType::Voice | ChannelType::Stage)) else {
|
||||
return Err(CoreError(format!("channel {channel_id} is not a voice channel")));
|
||||
};
|
||||
if let Some(g) = self.inner.settings.discord.guild_id {
|
||||
if gc.guild_id.get() != g {
|
||||
return Err(CoreError(format!("channel {channel_id} is not in the configured server")));
|
||||
}
|
||||
}
|
||||
if self.voice_channel_id() == Some(channel_id) {
|
||||
return Ok(());
|
||||
}
|
||||
self.leave_locked("moved").await;
|
||||
let call = self.inner.songbird.join(gc.guild_id, gc.id).await.map_err(|e| {
|
||||
CoreError(format!("could not connect to voice channel {channel_id}: {e}"))
|
||||
});
|
||||
let call = match call {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
let _ = self.inner.songbird.remove(gc.guild_id).await;
|
||||
return Err(e);
|
||||
}
|
||||
};
|
||||
let rx = Arc::new(Mutex::new(RxState::default()));
|
||||
{
|
||||
let mut c = call.lock().await;
|
||||
let h = Receiver { core: Arc::downgrade(&self.inner), rx: rx.clone(), guild_id: gc.guild_id.get() };
|
||||
c.add_global_event(CoreEvent_::SpeakingStateUpdate.into(), h.clone());
|
||||
c.add_global_event(CoreEvent_::RtpPacket.into(), h.clone());
|
||||
c.add_global_event(CoreEvent_::ClientDisconnect.into(), h.clone());
|
||||
c.add_global_event(CoreEvent_::DriverDisconnect.into(), h);
|
||||
}
|
||||
let alive = Arc::new(AtomicBool::new(true));
|
||||
tokio::spawn(speaking_end_timer(Arc::downgrade(&self.inner), rx.clone(), alive.clone()));
|
||||
*self.inner.session.lock().unwrap() = Some(Session { guild_id: gc.guild_id.get(), channel_id, rx, alive });
|
||||
let _ = self.inner.player.send(PlayerCmd::SetCall(Some(call)));
|
||||
debug_log!("voice", "joined voice {channel_id}");
|
||||
let _ = self.inner.events.send(CoreEvent::Joined { guild_id: gc.guild_id.get(), channel_id });
|
||||
let notice = CreateMessage::new().content(&self.inner.settings.voice.join_notice)
|
||||
.allowed_mentions(CreateAllowedMentions::new());
|
||||
if let Err(e) = gc.id.send_message(&http, notice).await {
|
||||
error_log!("voice", "could not post the transcription notice in {channel_id}: {e}");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Leave the voice channel. False if not in one.
|
||||
pub async fn leave(&self, reason: &str) -> bool {
|
||||
let _guard = self.inner.voice_lock.lock().await;
|
||||
self.leave_locked(reason).await
|
||||
}
|
||||
|
||||
async fn leave_locked(&self, reason: &str) -> bool {
|
||||
let Some(s) = self.inner.session.lock().unwrap().take() else { return false };
|
||||
end_session(&self.inner, s, reason).await;
|
||||
true
|
||||
}
|
||||
|
||||
/// Speak text in the voice channel. The request id, or None when not in a voice channel.
|
||||
pub fn say(&self, text: &str, more: bool) -> Option<String> {
|
||||
self.voice_channel_id()?;
|
||||
let id = with_playback(&self.inner, |p| p.say(text, more));
|
||||
Some(id)
|
||||
}
|
||||
|
||||
/// More text for a request started with `say(.., true)`.
|
||||
pub fn continue_text(&self, id: &str, text: &str, more: bool) {
|
||||
with_playback(&self.inner, |p| p.continue_text(id, text, more));
|
||||
}
|
||||
|
||||
/// Stop speaking at once and drop everything queued. How many requests were stopped.
|
||||
pub fn stop(&self) -> usize {
|
||||
with_playback(&self.inner, |p| p.stop())
|
||||
}
|
||||
|
||||
pub fn speaking(&self) -> bool {
|
||||
self.inner.playback.lock().unwrap().busy()
|
||||
}
|
||||
|
||||
// --- chat channels ----------------------------------------------------------------------------
|
||||
|
||||
/// The latest `limit` (1-100) messages of a registered channel, oldest first.
|
||||
pub async fn read_chat(&self, channel_id: u64, limit: u8) -> Result<Vec<ChatMessage>, CoreError> {
|
||||
self.inner.chat.check_read(channel_id)?;
|
||||
if !self.chat_content() {
|
||||
return Err(CoreError("the bot's Message Content Intent is off".into()));
|
||||
}
|
||||
let http = self.http()?;
|
||||
let mut msgs = ChannelId::new(channel_id)
|
||||
.messages(http.as_ref(), GetMessages::new().limit(limit.clamp(1, 100)))
|
||||
.await
|
||||
.map_err(cerr("could not read the channel"))?;
|
||||
msgs.sort_by_key(|m| m.id);
|
||||
Ok(msgs.iter().map(|m| to_chat(&self.inner, m)).collect())
|
||||
}
|
||||
|
||||
/// Send text to a registered channel, split at Discord's length limit. The message ids.
|
||||
pub async fn send_chat(&self, channel_id: u64, text: &str) -> Result<Vec<u64>, CoreError> {
|
||||
self.inner.chat.check_send(channel_id)?;
|
||||
if text.trim().is_empty() {
|
||||
return Err(CoreError("nothing to send".into()));
|
||||
}
|
||||
let http = self.http()?;
|
||||
let mut ids = vec![];
|
||||
for part in split_message(text, MESSAGE_LIMIT) {
|
||||
// Never ping anyone from text the assistant wrote.
|
||||
let m = CreateMessage::new().content(part).allowed_mentions(CreateAllowedMentions::new());
|
||||
let sent = ChannelId::new(channel_id).send_message(http.as_ref(), m).await.map_err(cerr("could not send"))?;
|
||||
ids.push(sent.id.get());
|
||||
}
|
||||
debug_log!("chat", "chat sent to {channel_id} parts={} chars={}", ids.len(), text.chars().count());
|
||||
Ok(ids)
|
||||
}
|
||||
|
||||
pub async fn close(&self) {
|
||||
self.leave("shutdown").await;
|
||||
if let Some(s) = self.inner.shards.get() {
|
||||
s.shutdown_all().await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
use songbird::CoreEvent as CoreEvent_;
|
||||
|
||||
fn with_playback<T>(inner: &Inner, f: impl FnOnce(&mut Playback<WorkerClient, PlayerSink>) -> T) -> T {
|
||||
let (out, events) = {
|
||||
let mut p = inner.playback.lock().unwrap();
|
||||
let out = f(&mut p);
|
||||
(out, p.take_events())
|
||||
};
|
||||
for e in events {
|
||||
let _ = inner.events.send(match e {
|
||||
PlaybackEvent::Start(id) => CoreEvent::SpeakStart(id),
|
||||
PlaybackEvent::End(e) => CoreEvent::SpeakEnd(e),
|
||||
});
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
async fn end_session(inner: &Arc<Inner>, s: Session, reason: &str) {
|
||||
s.alive.store(false, Ordering::Relaxed);
|
||||
with_playback(inner, |p| p.stop());
|
||||
let _ = inner.player.send(PlayerCmd::SetCall(None));
|
||||
let (heard, lost, late, bad) = {
|
||||
let mut rx = s.rx.lock().unwrap();
|
||||
let (l, t, b) = rx.streams.values().fold((0, 0, 0), |a, x| (a.0 + x.lost, a.1 + x.late, a.2 + x.bad));
|
||||
(std::mem::take(&mut rx.heard), l, t, b)
|
||||
};
|
||||
for user in heard {
|
||||
inner.worker.speaker_leave(&user.to_string());
|
||||
}
|
||||
inner.gate.lock().unwrap().clear();
|
||||
let _ = inner.songbird.remove(GuildId::new(s.guild_id)).await;
|
||||
debug_log!("voice", "left voice {} reason={reason} packets lost={lost} late={late} bad={bad}", s.channel_id);
|
||||
let _ = inner.events.send(CoreEvent::Left { guild_id: s.guild_id, channel_id: s.channel_id, reason: reason.into() });
|
||||
}
|
||||
|
||||
fn should_hear(inner: &Inner, user: u64) -> bool {
|
||||
if Some(user) == inner.bot_id.get().copied() {
|
||||
return false;
|
||||
}
|
||||
if inner.settings.voice.listen_to_bots {
|
||||
return true;
|
||||
}
|
||||
// Unknown users are people: the voice state that put a bot in the channel would have cached it.
|
||||
!inner.cache.get().and_then(|c| c.user(UserId::new(user)).map(|u| u.bot)).unwrap_or(false)
|
||||
}
|
||||
|
||||
fn name_of(inner: &Inner, user: u64) -> String {
|
||||
if let Some(n) = inner.names.lock().unwrap().get(&user) {
|
||||
return n.clone();
|
||||
}
|
||||
let guild = inner.session.lock().unwrap().as_ref().map(|s| s.guild_id).or(inner.settings.discord.guild_id);
|
||||
let name = inner.cache.get().and_then(|c| {
|
||||
guild
|
||||
.and_then(|g| c.guild(GuildId::new(g))?.members.get(&UserId::new(user)).map(|m| m.display_name().to_owned()))
|
||||
.or_else(|| c.user(UserId::new(user)).map(|u| u.display_name().to_owned()))
|
||||
});
|
||||
match name {
|
||||
Some(n) => {
|
||||
inner.names.lock().unwrap().insert(user, n.clone());
|
||||
n
|
||||
}
|
||||
None => user.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn heard(inner: &Inner, speaker: &str, utt: u32) -> Option<Heard> {
|
||||
let id: u64 = speaker.parse().ok()?;
|
||||
Some(Heard { speaker_id: id, speaker_name: name_of(inner, id), utt })
|
||||
}
|
||||
|
||||
fn to_chat(inner: &Inner, m: &Message) -> ChatMessage {
|
||||
let name = m.member.as_ref().and_then(|p| p.nick.clone())
|
||||
.unwrap_or_else(|| m.author.display_name().to_owned());
|
||||
ChatMessage {
|
||||
channel_id: m.channel_id.get(),
|
||||
message_id: m.id.get(),
|
||||
author_id: m.author.id.get(),
|
||||
author_name: name,
|
||||
author_is_bot: m.author.bot,
|
||||
from_self: Some(m.author.id.get()) == inner.bot_id.get().copied(),
|
||||
content: m.content.clone(),
|
||||
created_at: m.timestamp.to_rfc3339().unwrap_or_default(),
|
||||
}
|
||||
}
|
||||
|
||||
async fn worker_task(core: std::sync::Weak<Inner>, mut rx: mpsc::UnboundedReceiver<WorkerEvent>) {
|
||||
while let Some(ev) = rx.recv().await {
|
||||
let Some(inner) = core.upgrade() else { return };
|
||||
match ev {
|
||||
WorkerEvent::SpeechStart(m) => {
|
||||
if let Some(h) = heard(&inner, &m.speaker, m.utt) {
|
||||
let _ = inner.events.send(CoreEvent::SpeechStart(h));
|
||||
}
|
||||
}
|
||||
WorkerEvent::Partial(m) => {
|
||||
if let Some(h) = heard(&inner, &m.speaker, m.utt) {
|
||||
let _ = inner.events.send(CoreEvent::Partial(h, m.text));
|
||||
}
|
||||
}
|
||||
WorkerEvent::Final(m) => on_final(&inner, m),
|
||||
WorkerEvent::TtsAudio { id, seq, pcm } => with_playback(&inner, |p| p.audio(&id, seq, pcm)),
|
||||
WorkerEvent::TtsDone(d) => with_playback(&inner, |p| p.done(&d)),
|
||||
WorkerEvent::Ready(_) | WorkerEvent::Disconnected | WorkerEvent::Error(_) => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn on_final(inner: &Inner, m: Final) {
|
||||
debug_log!("voice", "final speaker={} utt={} reason={} chars={} audio_ms={} wait_ms={}", m.speaker, m.utt,
|
||||
m.end_reason, m.text.chars().count(), m.audio_ms, m.wait_ms);
|
||||
if let Some(h) = heard(inner, &m.speaker, m.utt) {
|
||||
let _ = inner.events.send(CoreEvent::Final(HeardFinal { heard: h, text: m.text, end_reason: m.end_reason,
|
||||
audio_ms: m.audio_ms, wait_ms: m.wait_ms, stt_ms: m.stt_ms }));
|
||||
}
|
||||
}
|
||||
|
||||
async fn player_task(core: std::sync::Weak<Inner>, mut rx: mpsc::UnboundedReceiver<PlayerCmd>) {
|
||||
let mut call: Option<Arc<tokio::sync::Mutex<Call>>> = None;
|
||||
let mut current: Option<(TrackHandle, u64)> = None;
|
||||
let mut generation = 0u64;
|
||||
while let Some(cmd) = rx.recv().await {
|
||||
let Some(inner) = core.upgrade() else { return };
|
||||
match cmd {
|
||||
PlayerCmd::SetCall(c) => {
|
||||
if let Some((t, _)) = current.take() {
|
||||
let _ = t.stop();
|
||||
}
|
||||
call = c;
|
||||
}
|
||||
PlayerCmd::Play(pcm) => {
|
||||
generation += 1;
|
||||
let Some(c) = &call else {
|
||||
// Nowhere to play: count it as played so the queue moves on.
|
||||
with_playback(&inner, |p| p.clip_ended());
|
||||
continue;
|
||||
};
|
||||
let track = c.lock().await.play_only_input(Input::from(wav(&pcm)));
|
||||
let ended = TrackEnded { tx: inner.player.clone(), generation };
|
||||
let _ = track.add_event(Event::Track(TrackEvent::End), ended);
|
||||
let ended = TrackEnded { tx: inner.player.clone(), generation };
|
||||
let _ = track.add_event(Event::Track(TrackEvent::Error), ended);
|
||||
current = Some((track, generation));
|
||||
}
|
||||
PlayerCmd::Stop => {
|
||||
if let Some((t, _)) = current.take() {
|
||||
let _ = t.stop();
|
||||
}
|
||||
}
|
||||
PlayerCmd::Ended(g) => {
|
||||
// Only the end of the clip we started counts; a stop() already moved on.
|
||||
if current.as_ref().is_some_and(|(_, cg)| *cg == g) {
|
||||
current = None;
|
||||
with_playback(&inner, |p| p.clip_ended());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- receive --------------------------------------------------------------------------------------
|
||||
|
||||
#[derive(Clone)]
|
||||
struct Receiver {
|
||||
core: std::sync::Weak<Inner>,
|
||||
rx: Arc<Mutex<RxState>>,
|
||||
guild_id: u64,
|
||||
}
|
||||
|
||||
impl Receiver {
|
||||
fn packet(&self, inner: &Inner, rtp: &songbird::events::context_data::RtpData) {
|
||||
let Some((seq, ssrc, opus)) = opus_payload(&rtp.packet, rtp.payload_offset, rtp.payload_end_pad) else { return };
|
||||
let mut rx = self.rx.lock().unwrap();
|
||||
let Some(&user) = rx.ssrc_user.get(&ssrc) else { return };
|
||||
if !should_hear(inner, user) {
|
||||
return;
|
||||
}
|
||||
let mut gate = inner.gate.lock().unwrap();
|
||||
if rx.talking.insert(user, Instant::now()).is_none() && !gate.start(user) {
|
||||
debug_log!("voice", "speaker {user} not heard: {} already speaking", gate.count());
|
||||
}
|
||||
// Decode even when not heard, so the decoder state stays continuous.
|
||||
let frames = rx.streams.entry(ssrc).or_insert_with(|| SpeakerStream::new(OpusFrames::new())).push(seq, opus.to_vec());
|
||||
if gate.is_active(user) && !frames.is_empty() {
|
||||
rx.heard.insert(user);
|
||||
let id = user.to_string();
|
||||
for f in frames {
|
||||
inner.worker.send_audio(&id, &f);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn left(&self, inner: &Inner, user: u64) {
|
||||
let mut rx = self.rx.lock().unwrap();
|
||||
rx.talking.remove(&user);
|
||||
let ssrcs: Vec<u32> = rx.ssrc_user.iter().filter(|(_, u)| **u == user).map(|(s, _)| *s).collect();
|
||||
for ssrc in ssrcs {
|
||||
rx.ssrc_user.remove(&ssrc);
|
||||
rx.streams.remove(&ssrc);
|
||||
}
|
||||
let was_heard = inner.gate.lock().unwrap().end(user);
|
||||
if rx.heard.remove(&user) || was_heard {
|
||||
inner.worker.speaker_leave(&user.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl VoiceEventHandler for Receiver {
|
||||
async fn act(&self, ctx: &EventContext<'_>) -> Option<Event> {
|
||||
let inner = self.core.upgrade()?;
|
||||
match ctx {
|
||||
EventContext::SpeakingStateUpdate(s) => {
|
||||
if let Some(u) = s.user_id {
|
||||
self.rx.lock().unwrap().ssrc_user.insert(s.ssrc, u.0);
|
||||
}
|
||||
}
|
||||
EventContext::RtpPacket(p) => self.packet(&inner, p),
|
||||
EventContext::ClientDisconnect(d) => self.left(&inner, d.user_id.0),
|
||||
EventContext::DriverDisconnect(_) => {
|
||||
// The driver gave up reconnecting (kicked, channel deleted, long network loss).
|
||||
let core = DiscordCore { inner: inner.clone() };
|
||||
let guild = self.guild_id;
|
||||
tokio::spawn(async move {
|
||||
let _g = core.inner.voice_lock.lock().await;
|
||||
let s = {
|
||||
let mut s = core.inner.session.lock().unwrap();
|
||||
if s.as_ref().is_some_and(|s| s.guild_id == guild) { s.take() } else { None }
|
||||
};
|
||||
if let Some(s) = s {
|
||||
end_session(&core.inner, s, "disconnected").await;
|
||||
}
|
||||
});
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Every 20 ms: anyone without a packet for SPEAKING_END has stopped. Play out what their reorder
|
||||
/// window still holds, then tell the worker.
|
||||
async fn speaking_end_timer(core: std::sync::Weak<Inner>, rx: Arc<Mutex<RxState>>, alive: Arc<AtomicBool>) {
|
||||
let mut every = tokio::time::interval(Duration::from_millis(20));
|
||||
while alive.load(Ordering::Relaxed) {
|
||||
every.tick().await;
|
||||
let Some(inner) = core.upgrade() else { return };
|
||||
let mut rx = rx.lock().unwrap();
|
||||
let now = Instant::now();
|
||||
let ended: Vec<u64> = rx.talking.iter().filter(|(_, t)| now.duration_since(**t) >= SPEAKING_END).map(|(u, _)| *u).collect();
|
||||
for user in ended {
|
||||
rx.talking.remove(&user);
|
||||
let ssrcs: Vec<u32> = rx.ssrc_user.iter().filter(|(_, u)| **u == user).map(|(s, _)| *s).collect();
|
||||
let mut gate = inner.gate.lock().unwrap();
|
||||
let active = gate.is_active(user);
|
||||
let id = user.to_string();
|
||||
for ssrc in ssrcs {
|
||||
if let Some(st) = rx.streams.get_mut(&ssrc) {
|
||||
for f in st.flush() {
|
||||
if active {
|
||||
inner.worker.send_audio(&id, &f);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if gate.end(user) {
|
||||
inner.worker.speaking_end(&id);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- gateway events -------------------------------------------------------------------------------
|
||||
|
||||
struct Handler {
|
||||
core: DiscordCore,
|
||||
ready: mpsc::UnboundedSender<Ready>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl EventHandler for Handler {
|
||||
async fn ready(&self, _: Context, ready: Ready) {
|
||||
let _ = self.ready.send(ready);
|
||||
}
|
||||
|
||||
async fn voice_state_update(&self, _: Context, old: Option<VoiceState>, new: VoiceState) {
|
||||
let inner = &self.core.inner;
|
||||
let user = new.user_id.get();
|
||||
if let Some(m) = &new.member {
|
||||
inner.names.lock().unwrap().insert(user, m.display_name().to_owned());
|
||||
}
|
||||
let (session_channel, rx) = {
|
||||
let s = inner.session.lock().unwrap();
|
||||
match s.as_ref() {
|
||||
Some(s) if new.guild_id.map(|g| g.get()) == Some(s.guild_id) => (s.channel_id, s.rx.clone()),
|
||||
_ => return,
|
||||
}
|
||||
};
|
||||
let new_channel = new.channel_id.map(|c| c.get());
|
||||
if Some(user) == inner.bot_id.get().copied() {
|
||||
// Moved by someone else: follow the new channel id. Disconnects end via DriverDisconnect.
|
||||
if let Some(c) = new_channel.filter(|c| *c != session_channel) {
|
||||
if let Some(s) = inner.session.lock().unwrap().as_mut() {
|
||||
s.channel_id = c;
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
let was_here = old.and_then(|o| o.channel_id).map(|c| c.get()) == Some(session_channel);
|
||||
if was_here && new_channel != Some(session_channel) {
|
||||
Receiver { core: Arc::downgrade(inner), rx, guild_id: 0 }.left(inner, user);
|
||||
}
|
||||
}
|
||||
|
||||
async fn message(&self, _: Context, m: Message) {
|
||||
let inner = &self.core.inner;
|
||||
let ch = m.channel_id.get();
|
||||
if !self.core.chat_content() || !inner.chat.can_read(ch) || Some(m.author.id.get()) == self.core.bot_id() {
|
||||
return;
|
||||
}
|
||||
debug_log!("chat", "chat message in {ch} from {} chars={}", m.author.id, m.content.chars().count());
|
||||
let _ = inner.events.send(CoreEvent::Chat(to_chat(inner, &m)));
|
||||
}
|
||||
|
||||
async fn interaction_create(&self, _: Context, interaction: Interaction) {
|
||||
if let Interaction::Command(cmd) = interaction {
|
||||
self.on_command(cmd).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Handler {
|
||||
async fn on_command(&self, cmd: CommandInteraction) {
|
||||
let inner = &self.core.inner;
|
||||
if cmd.data.name != inner.settings.discord.command_name {
|
||||
return;
|
||||
}
|
||||
let Some(guild) = cmd.guild_id else { return };
|
||||
if inner.settings.discord.guild_id.is_some_and(|g| g != guild.get()) {
|
||||
return;
|
||||
}
|
||||
let Ok(http) = self.core.http().cloned() else { return };
|
||||
let eph = inner.settings.discord.ephemeral_replies;
|
||||
let sub = cmd.data.options.first().filter(|o| matches!(o.value, CommandDataOptionValue::SubCommand(_)))
|
||||
.map(|o| o.name.clone()).unwrap_or_default();
|
||||
debug_log!("discord", "command {sub} from {}", cmd.user.id);
|
||||
let reply = |content: &str| CreateInteractionResponse::Message(
|
||||
CreateInteractionResponseMessage::new().content(content).ephemeral(eph));
|
||||
let result: Result<(), CoreError> = async {
|
||||
match sub.as_str() {
|
||||
"join" => {
|
||||
let channel = inner.cache.get().and_then(|c| {
|
||||
c.guild(guild).and_then(|g| g.voice_states.get(&cmd.user.id).and_then(|v| v.channel_id))
|
||||
});
|
||||
let Some(channel) = channel else {
|
||||
cmd.create_response(&http, reply("먼저 음성 채널에 들어간 뒤 다시 불러 주세요.")).await
|
||||
.map_err(cerr("reply"))?;
|
||||
return Ok(());
|
||||
};
|
||||
cmd.create_response(&http, CreateInteractionResponse::Defer(
|
||||
CreateInteractionResponseMessage::new().ephemeral(eph))).await.map_err(cerr("defer"))?;
|
||||
self.core.join(channel.get()).await?;
|
||||
cmd.edit_response(&http, EditInteractionResponse::new().content("음성 채널에 들어왔습니다.")).await
|
||||
.map_err(cerr("reply"))?;
|
||||
}
|
||||
"leave" => {
|
||||
let left = self.core.leave("command").await;
|
||||
cmd.create_response(&http, reply(if left { "음성 채널에서 나갔습니다." } else { "음성 채널에 있지 않습니다." }))
|
||||
.await.map_err(cerr("reply"))?;
|
||||
}
|
||||
"stop" => {
|
||||
let stopped = self.core.stop();
|
||||
cmd.create_response(&http, reply(if stopped > 0 { "멈췄습니다." } else { "지금은 말하고 있지 않습니다." }))
|
||||
.await.map_err(cerr("reply"))?;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
error_log!("discord", "command {sub} failed: {e}");
|
||||
let content = "처리하지 못했습니다. 잠시 뒤 다시 시도해 주세요.";
|
||||
if cmd.edit_response(&http, EditInteractionResponse::new().content(content)).await.is_err() {
|
||||
let _ = cmd.create_response(&http, reply(content)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn wav_header_describes_48k_stereo_pcm16() {
|
||||
let w = super::wav(&[1, 2, 3, 4]);
|
||||
assert_eq!(&w[0..4], b"RIFF");
|
||||
assert_eq!(u32::from_le_bytes(w[4..8].try_into().unwrap()), 40);
|
||||
assert_eq!(u16::from_le_bytes(w[22..24].try_into().unwrap()), 2);
|
||||
assert_eq!(u32::from_le_bytes(w[24..28].try_into().unwrap()), 48000);
|
||||
assert_eq!(u32::from_le_bytes(w[40..44].try_into().unwrap()), 4);
|
||||
assert_eq!(&w[44..], &[1, 2, 3, 4]);
|
||||
}
|
||||
|
||||
}
|
||||
182
app/src/discord_core.spec.md
Normal file
182
app/src/discord_core.spec.md
Normal file
@@ -0,0 +1,182 @@
|
||||
# Discord voice core spec (v2 M2)
|
||||
|
||||
The Discord voice core is the part of the Windows app (Rust; the GUI arrives in M5 as a Tauri 2 shell)
|
||||
that connects the bot to Discord: it joins a voice channel, hears each person separately, streams their
|
||||
audio to the voice worker for transcription, plays the worker's speech back into the channel, and reads
|
||||
and writes the registered chat channels. It has no assistant logic. It reports what it heard and does
|
||||
what it is told; the brain (M3) and call detection and response policies (M4) sit on top of it.
|
||||
Design background: `docs/v2/architecture.md`, `docs/v2/conversation.md`, `docs/v2/privacy.md`. The
|
||||
worker side of the link is `src/voice_worker/voice_worker.spec.md`.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Only where the bot is.** The bot hears only the voice channel it is in and reads only the chat
|
||||
channels registered in settings with reading on. Every join posts the transcription notice in the
|
||||
voice channel's chat.
|
||||
- **Chat is data.** Messages read from chat channels are handed on as data, never acted on as
|
||||
commands. Text the bot sends never pings anyone (empty allowed mentions).
|
||||
- **No text in logs.** Logs carry ids, sizes, counts and timings, never what anyone said or wrote, or
|
||||
what the bot says. Debug logs (`debug_log!`, on with `JARVIS_DEBUG=1`) go at joins and leaves,
|
||||
command use, finals, speech requests and stops, chat events and sends, and worker connection changes.
|
||||
- **Language-agnostic.** Nothing in the core matches words. Wake words are only passed to the STT as a
|
||||
hint until call detection (M4) uses them.
|
||||
- **Testable without Discord.** The pieces with rules (receive window, playback queue, speaker limit,
|
||||
chat access and splitting, settings, worker link, worker process) are plain types tested with fakes;
|
||||
the Discord glue is tested on real Discord by an opt-in live test.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
app/
|
||||
├── Cargo.toml # crate jarvis-app: library + the jarvis-app binary
|
||||
├── src/main.rs # jarvis-app <data folder>: runs the services until Ctrl+C / SIGTERM
|
||||
├── src/services.rs # settings -> voice worker process -> worker client -> Discord core
|
||||
├── src/discord.rs # Discord client (serenity), voice (songbird), slash command, chat channels
|
||||
├── src/receive.rs # per-speaker reorder window and Opus decoding
|
||||
├── src/playback.rs # speech queue (TTS requests -> sentence clips -> player)
|
||||
├── src/speakers.rs # who is transcribed right now (max speakers)
|
||||
├── src/chat.rs # chat channel access rules and message splitting
|
||||
├── src/worker_client.rs # WebSocket client for the voice worker
|
||||
├── src/worker_process.rs # starts and restarts the Python voice worker
|
||||
├── src/frames.rs # binary frame layout shared with voice_worker/protocol.py
|
||||
├── src/settings.rs # settings.json loading and validation
|
||||
└── src/log.rs # debug_log! / error_log!
|
||||
```
|
||||
|
||||
## Starting
|
||||
|
||||
`start_services(data_dir)` reads `<data_dir>/config/settings.json`, starts the voice worker process,
|
||||
connects to it once it prints `READY`, and starts the Discord core. Until the setup wizard (M5) stores
|
||||
it encrypted, the bot token comes from `JARVIS_DISCORD_TOKEN` and the data folder from the first
|
||||
argument or `JARVIS_DATA_DIR`. The binary stops every service on Ctrl+C or SIGTERM.
|
||||
|
||||
## Settings
|
||||
|
||||
`<data folder>/config/settings.json`. Every key is optional; missing keys take the defaults. A bad
|
||||
value stops the app with a message naming the key. Unknown keys are ignored. Discord IDs are JSON
|
||||
strings (numbers would lose precision).
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `discord.guildId` | `null` | The one server. The slash command is registered there; commands and channels from other servers are ignored. |
|
||||
| `discord.voiceChannelId` | `null` | Join this voice channel on start ("자동 입장"). |
|
||||
| `discord.commandName` | `"자비스"` | Slash command name (Discord rule: 1-32 lowercase letters, digits, `-`, `_`). |
|
||||
| `discord.ephemeralReplies` | `true` | Command replies only the caller sees. |
|
||||
| `voice.maxSpeakers` | `10` | Most people transcribed at once, 1-10. |
|
||||
| `voice.listenToBots` | `false` | Also transcribe other bots (music bots are ignored by default). |
|
||||
| `voice.joinNotice` | "자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다." | Posted in the voice channel's chat on every join. Must not be empty. |
|
||||
| `voice.wakeWords` | `["자비스"]` | Sent to the worker as STT `hotwords`. |
|
||||
| `voice.stt` | `{}` | Other worker STT settings (`partials`, `end_silence_ms`, `final_beam`, `language`, ...); the worker validates them. |
|
||||
| `chatChannels` | `[]` | `{id, name, description, read (true), send (true)}` per channel. |
|
||||
| `voiceWorker.python` | `python3` (`python.exe` on Windows) | Python of the worker venv. |
|
||||
| `voiceWorker.cwd` | the repo's `src/` (development builds) | Folder holding the `voice_worker` package. |
|
||||
| `voiceWorker.args` | `[]` | Extra worker arguments, e.g. `["--melo-python", "<MeloTTS venv python>"]`. |
|
||||
|
||||
## Voice worker process
|
||||
|
||||
The app starts `python -m voice_worker --host 127.0.0.1 --port <port> <args>` in `voiceWorker.cwd`
|
||||
(no console window on Windows). Each launch makes a new random token (32 bytes) and passes it only in
|
||||
`JARVIS_VOICE_TOKEN`. The port is a free loopback port chosen once at start. If the worker exits, it is
|
||||
started again on the same port with the same token after 1 s, doubling to at most 30 s while it keeps
|
||||
failing (reset once a run printed `READY`). Stopping kills it; MeloTTS's child process ends when its
|
||||
input closes.
|
||||
|
||||
## Worker link
|
||||
|
||||
One WebSocket to the worker, `hello` with the token, the Discord PCM format both ways
|
||||
(48 kHz stereo), and the STT settings (`voice.stt` with `hotwords` = `voice.wakeWords`). If the
|
||||
connection drops, the client reconnects with backoff (0.5 s doubling to 5 s) and says hello again with
|
||||
the same settings, including changes made since with `configure()`. While disconnected, audio frames
|
||||
are dropped (not queued), and every TTS request in flight ends as failed, so playback moves on. A TTS
|
||||
request the worker refuses (`tts_unavailable`) also ends as failed. TTS audio for request ids the client
|
||||
did not open is ignored.
|
||||
|
||||
## Voice
|
||||
|
||||
One voice session at a time, in the configured server. Joining another channel leaves the current one.
|
||||
|
||||
- **Join** (slash command `join`, auto-join, or `join(channel_id)`): connect through songbird with DAVE
|
||||
end-to-end encryption, not deafened, then post `voice.joinNotice` in the voice channel's chat. A
|
||||
notice that cannot be posted (no permission) is logged and the bot stays.
|
||||
- **Hearing**: songbird only decrypts (transport and DAVE). Its own Opus decoding is not used: in
|
||||
songbird 0.6.1 it decodes the payload without trimming what DAVE decryption left at its end, which
|
||||
garbled, repeated or dropped about a third of the frames (measured with a test tone that changes
|
||||
pitch every 20 ms). The core takes each decrypted RTP packet with songbird's exact payload bounds:
|
||||
- Packets go through a per-speaker reorder window in sequence-number order. A missing packet is
|
||||
waited for until two later packets have arrived (about 40 ms), then rebuilt from the next packet's
|
||||
Opus in-band FEC (the last missing frame) or concealed (any earlier ones). Late and duplicate
|
||||
packets are dropped; a jump of 64 or more starts the window over.
|
||||
- Frames are decoded with libopus to 48 kHz stereo PCM and sent to the worker in 20 ms frames under
|
||||
the user's ID. Decoding goes on for people who are not heard, so their decoder stays continuous.
|
||||
- The bot's own audio is never heard; other bots are skipped unless `voice.listenToBots`.
|
||||
- At most `voice.maxSpeakers` people are transcribed at once. Someone who starts while the limit is
|
||||
reached is not heard until their next start after a place frees up.
|
||||
- No packet for 100 ms ends a person's speaking: what their window still holds is played out, then
|
||||
`speaking_end` is sent.
|
||||
- When a person leaves the channel (gateway voice state or songbird's client disconnect), the bot
|
||||
sends `speaker_leave`, so their utterance finishes and the worker frees their place.
|
||||
- **Events** for the layers above: `SpeechStart`, `Partial` and `Final` with the speaker's ID and
|
||||
server display name; `SpeakStart` and `SpeakEnd`; `Joined` and `Left` (with a reason: `command`,
|
||||
`moved`, `disconnected`, `shutdown`).
|
||||
- **Speaking**: `say(text, more)` queues a TTS request and returns its id (`None` when not in a voice
|
||||
channel); `continue_text(id, text, more)` adds text to an open request. Each sentence the worker
|
||||
sends becomes one clip (a 48 kHz stereo WAV played through songbird without resampling); clips play
|
||||
in sentence order as soon as they arrive, and requests play first come, first served. A sentence that
|
||||
failed is skipped and the rest still play. `SpeakEnd` reports how many sentences played and whether
|
||||
the request was cancelled or failed.
|
||||
- **Stop** (slash command `stop`, or `stop()`): the current clip stops at the next 20 ms mix, every
|
||||
queued request is dropped, and unfinished requests are cancelled at the worker; each ends with
|
||||
`cancelled: true`. Stopping by voice ("자비스, 멈춰") needs call detection and comes in M4.
|
||||
- **Leave** (slash command `leave`, or `leave()`): stop speaking, send `speaker_leave` for everyone
|
||||
heard, and disconnect. When songbird's driver gives up the connection (kicked, channel deleted, long
|
||||
network loss) the session ends with `disconnected`. When someone moves the bot to another channel,
|
||||
the session follows it.
|
||||
|
||||
## Slash command
|
||||
|
||||
One guild command, `/<commandName>` with subcommands `join` (the caller's voice channel), `leave`
|
||||
and `stop`, registered on every start with a create-or-update by name, so the app's other commands in
|
||||
the server are left alone. A failed registration (for example a bot invited without the
|
||||
`applications.commands` scope) is logged and the bot runs on without it. Usable only in the configured
|
||||
server. Replies are in Korean, ephemeral by default. Renaming `commandName` leaves the old command
|
||||
registered until it is removed (M5 setup).
|
||||
|
||||
## Chat channels
|
||||
|
||||
- The bot asks for the message intents only when some registered channel has reading on. Reading
|
||||
needs the Message Content intent, which is checked in the application's flags before login. When it
|
||||
is off, the bot still logs in, voice works, and an error explains how to turn it on; chat reads then
|
||||
fail with a clear error instead of returning empty messages.
|
||||
- **Live**: every new message in a readable registered channel, except the bot's own, is emitted as a
|
||||
`Chat` event: channel, message and author IDs, author display name, whether the author is a bot, the
|
||||
text, and the time.
|
||||
- **History**: `read_chat(channel_id, limit)` returns the latest 1-100 messages, oldest first, with
|
||||
`from_self` marking the bot's own.
|
||||
- **Send**: `send_chat(channel_id, text)` sends to a registered channel with sending on, split at
|
||||
Discord's 2,000-character limit at the last newline or space (never inside a character), and returns
|
||||
the message ids.
|
||||
- A channel that is not registered, or has the needed switch off, is refused with an access error.
|
||||
|
||||
## Tests
|
||||
|
||||
- `cargo test`: unit tests for the receive window (with a tagging fake and real libopus), playback
|
||||
order, gaps, stop and cancel, speaker limit, chat access and splitting, settings, frames; the worker
|
||||
link against a fake server (hello, events, reconnect, failed TTS); the worker process with a
|
||||
stand-in `voice_worker` package run by a real Python (token only in the environment, restart on the
|
||||
same port, stop).
|
||||
- `JARVIS_VOICE_PYTHON=<venv python> cargo test --test worker_protocol` runs the worker link and
|
||||
playback against the real Python worker server with fake engines (`tests/fixtures/fake_engine_worker.py`).
|
||||
- `tests/live_discord.rs` (opt-in, `JARVIS_LIVE_DISCORD=1`, see the file header): two bots in one
|
||||
voice channel with the real worker on the GPU. It refuses to run if a person is in the channel or
|
||||
either bot is already in a voice channel, and cleans up even when a step fails. It checks command
|
||||
registration, transcription of recorded sentences played into the channel, speech the other bot
|
||||
hears and the STT reads back, stop, chat events, history, sending without pings, and leaving.
|
||||
Measured on this server's RTX 5050 (2026-11-11): five sentences all heard (character error rate 0%
|
||||
for three, 5% and 12.5% for two), 0.71-0.90 s from the end of speech to the final, Jarvis's speech
|
||||
read back at 3.4%, sound gone 0.08 s after stop.
|
||||
|
||||
## Known risks
|
||||
|
||||
- songbird issue #310 (open): receive can stall after a DAVE (MLS) group change on some joins. The live
|
||||
test covers one change (the second bot joining); channels where many people come and go need a
|
||||
longer check before M4.
|
||||
73
app/src/frames.rs
Normal file
73
app/src/frames.rs
Normal file
@@ -0,0 +1,73 @@
|
||||
//! Binary frame layout shared with the Python voice worker (`src/voice_worker/protocol.py`).
|
||||
|
||||
pub const KIND_AUDIO: u8 = 0x01;
|
||||
pub const KIND_TTS: u8 = 0x02;
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub struct TtsFrame {
|
||||
pub id: String,
|
||||
pub seq: u32,
|
||||
pub pcm: Vec<u8>,
|
||||
}
|
||||
|
||||
/// `0x01 | u8 n | speaker id | PCM16LE` in the hello "audio" format. None if the id does not fit.
|
||||
pub fn encode_audio_frame(speaker: &str, pcm: &[i16]) -> Option<Vec<u8>> {
|
||||
let id = speaker.as_bytes();
|
||||
if id.is_empty() || id.len() > 255 {
|
||||
return None;
|
||||
}
|
||||
let mut out = Vec::with_capacity(2 + id.len() + pcm.len() * 2);
|
||||
out.push(KIND_AUDIO);
|
||||
out.push(id.len() as u8);
|
||||
out.extend_from_slice(id);
|
||||
for s in pcm {
|
||||
out.extend_from_slice(&s.to_le_bytes());
|
||||
}
|
||||
Some(out)
|
||||
}
|
||||
|
||||
/// `0x02 | u8 n | request id | u32 LE seq | PCM16LE` in the hello "tts" format. None if malformed.
|
||||
pub fn decode_tts_frame(frame: &[u8]) -> Option<TtsFrame> {
|
||||
if frame.len() < 2 || frame[0] != KIND_TTS {
|
||||
return None;
|
||||
}
|
||||
let n = frame[1] as usize;
|
||||
let end = 2 + n;
|
||||
if n == 0 || frame.len() < end + 4 {
|
||||
return None;
|
||||
}
|
||||
let id = std::str::from_utf8(&frame[2..end]).ok()?.to_owned();
|
||||
let seq = u32::from_le_bytes(frame[end..end + 4].try_into().ok()?);
|
||||
Some(TtsFrame { id, seq, pcm: frame[end + 4..].to_vec() })
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn audio_frame_is_kind_id_length_id_then_pcm() {
|
||||
let f = encode_audio_frame("유저", &[1, -2]).unwrap();
|
||||
let id = "유저".as_bytes();
|
||||
let mut want = vec![KIND_AUDIO, id.len() as u8];
|
||||
want.extend_from_slice(id);
|
||||
want.extend_from_slice(&[1, 0, 0xfe, 0xff]);
|
||||
assert_eq!(f, want);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ids_must_fit_in_a_byte() {
|
||||
assert!(encode_audio_frame("", &[0]).is_none());
|
||||
assert!(encode_audio_frame(&"x".repeat(256), &[0]).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tts_frames_decode_and_truncated_ones_are_refused() {
|
||||
let mut f = vec![KIND_TTS, 2, b'r', b'1'];
|
||||
f.extend_from_slice(&7u32.to_le_bytes());
|
||||
f.extend_from_slice(&[9, 9]);
|
||||
assert_eq!(decode_tts_frame(&f), Some(TtsFrame { id: "r1".into(), seq: 7, pcm: vec![9, 9] }));
|
||||
assert_eq!(decode_tts_frame(&f[..5]), None);
|
||||
assert_eq!(decode_tts_frame(&[KIND_AUDIO, 1, b'a', 0, 0, 0, 0]), None);
|
||||
}
|
||||
}
|
||||
13
app/src/lib.rs
Normal file
13
app/src/lib.rs
Normal file
@@ -0,0 +1,13 @@
|
||||
//! Jarvis v2 app core (see `app/src/discord_core.spec.md`).
|
||||
|
||||
pub mod log;
|
||||
pub mod chat;
|
||||
pub mod discord;
|
||||
pub mod frames;
|
||||
pub mod playback;
|
||||
pub mod receive;
|
||||
pub mod services;
|
||||
pub mod settings;
|
||||
pub mod speakers;
|
||||
pub mod worker_client;
|
||||
pub mod worker_process;
|
||||
43
app/src/log.rs
Normal file
43
app/src/log.rs
Normal file
@@ -0,0 +1,43 @@
|
||||
//! Debug logging, the Rust side of `src/jarvis/debug.py`.
|
||||
//!
|
||||
//! On only when `JARVIS_DEBUG=1`. Messages carry ids, sizes, counts and timings, never what anyone
|
||||
//! said or wrote (no transcripts, no chat text, no reply text).
|
||||
|
||||
use std::sync::OnceLock;
|
||||
|
||||
pub fn debug_enabled() -> bool {
|
||||
static ON: OnceLock<bool> = OnceLock::new();
|
||||
*ON.get_or_init(|| std::env::var("JARVIS_DEBUG").as_deref() == Ok("1"))
|
||||
}
|
||||
|
||||
/// `[ category ] message`, the category centred in 10 columns like the Python `debug_log`.
|
||||
pub fn line(category: &str, message: &str) -> String {
|
||||
format!("[{category:^10}] {message}")
|
||||
}
|
||||
|
||||
#[macro_export]
|
||||
macro_rules! debug_log {
|
||||
($cat:expr, $($arg:tt)*) => {
|
||||
if $crate::log::debug_enabled() {
|
||||
eprintln!("{}", $crate::log::line($cat, &format!($($arg)*)));
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/// Errors are always printed; the same rule applies: no transcript or chat text in the message.
|
||||
#[macro_export]
|
||||
macro_rules! error_log {
|
||||
($cat:expr, $($arg:tt)*) => {
|
||||
eprintln!("[{}] ERROR {}", $cat, format!($($arg)*));
|
||||
};
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn category_is_centred_like_python() {
|
||||
// Python: f"[{'voice':^10}]" == "[ voice ]"
|
||||
assert_eq!(super::line("voice", "x"), "[ voice ] x");
|
||||
assert_eq!(super::line("worker", "x"), "[ worker ] x");
|
||||
}
|
||||
}
|
||||
49
app/src/main.rs
Normal file
49
app/src/main.rs
Normal file
@@ -0,0 +1,49 @@
|
||||
//! Jarvis app (headless until the Tauri GUI arrives in M5): jarvis-app <data folder>
|
||||
//! The data folder can also come from JARVIS_DATA_DIR; the bot token from JARVIS_DISCORD_TOKEN.
|
||||
|
||||
use jarvis_app::discord::CoreEvent;
|
||||
use jarvis_app::services::start_services;
|
||||
use jarvis_app::{debug_log, error_log};
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let data_dir = std::env::args().nth(1).or_else(|| std::env::var("JARVIS_DATA_DIR").ok());
|
||||
let Some(data_dir) = data_dir else {
|
||||
error_log!("app", "usage: jarvis-app <data folder> (or set JARVIS_DATA_DIR)");
|
||||
std::process::exit(2);
|
||||
};
|
||||
let mut services = match start_services(std::path::Path::new(&data_dir)).await {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
error_log!("app", "could not start: {e}");
|
||||
std::process::exit(1);
|
||||
}
|
||||
};
|
||||
loop {
|
||||
tokio::select! {
|
||||
ev = services.events.recv() => match ev {
|
||||
// The brain (M3) consumes these; for now only their shape is logged, never the text.
|
||||
Some(CoreEvent::Final(f)) => debug_log!("app", "heard {} utt={} chars={}", f.heard.speaker_id, f.heard.utt, f.text.chars().count()),
|
||||
Some(_) => {}
|
||||
None => break,
|
||||
},
|
||||
_ = shutdown_signal() => break,
|
||||
}
|
||||
}
|
||||
services.stop().await;
|
||||
}
|
||||
|
||||
async fn shutdown_signal() {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let mut term = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()).expect("signal");
|
||||
tokio::select! {
|
||||
_ = tokio::signal::ctrl_c() => {}
|
||||
_ = term.recv() => {}
|
||||
}
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
let _ = tokio::signal::ctrl_c().await;
|
||||
}
|
||||
}
|
||||
@@ -1,36 +0,0 @@
|
||||
/**
|
||||
* Electron main process. The GUI arrives in M5; for now the app runs its services in the background.
|
||||
* The data folder comes from JARVIS_DATA_DIR until the setup wizard picks one (M5).
|
||||
*/
|
||||
import { join } from "node:path";
|
||||
import { app } from "electron";
|
||||
import { errorLog } from "./core/log.js";
|
||||
import { startServices, type Services } from "./services.js";
|
||||
|
||||
const dataDir = process.env.JARVIS_DATA_DIR;
|
||||
// Electron's own profile goes inside the data folder too, not into the user's AppData.
|
||||
if (dataDir) app.setPath("userData", join(dataDir, "cache", "electron"));
|
||||
|
||||
if (!app.requestSingleInstanceLock()) app.quit();
|
||||
|
||||
let services: Services | null = null;
|
||||
|
||||
app.on("window-all-closed", () => {
|
||||
// No windows yet; keep running.
|
||||
});
|
||||
|
||||
app.whenReady().then(async () => {
|
||||
if (!dataDir) throw new Error("JARVIS_DATA_DIR is not set");
|
||||
services = await startServices(dataDir);
|
||||
}).catch((e: Error) => {
|
||||
errorLog(`could not start: ${e.message}`);
|
||||
app.exit(1);
|
||||
});
|
||||
|
||||
let quitting = false;
|
||||
app.on("before-quit", (event) => {
|
||||
if (quitting || !services) return;
|
||||
event.preventDefault();
|
||||
quitting = true;
|
||||
void services.stop().finally(() => app.quit());
|
||||
});
|
||||
315
app/src/playback.rs
Normal file
315
app/src/playback.rs
Normal file
@@ -0,0 +1,315 @@
|
||||
//! The bot's mouth: turns text into queued speech, one request after another.
|
||||
//!
|
||||
//! Each request goes to the voice worker as TTS; the worker sends back one PCM chunk per sentence.
|
||||
//! Chunks play in order as soon as they arrive, so the first sentence plays while later ones are still
|
||||
//! being made. Requests play first come, first served. `stop()` silences everything at once and
|
||||
//! cancels what the worker has not made yet.
|
||||
|
||||
use std::collections::{BTreeMap, VecDeque};
|
||||
|
||||
use crate::worker_client::TtsDone;
|
||||
|
||||
pub trait TtsBackend {
|
||||
/// Send text for request `id`; false when the worker is not connected (the request then fails).
|
||||
fn tts(&self, id: &str, text: &str, more: bool) -> bool;
|
||||
fn tts_cancel(&self, id: &str);
|
||||
}
|
||||
|
||||
/// Plays one PCM clip at a time; its owner calls `Playback::clip_ended()` when the clip finishes.
|
||||
pub trait ClipSink {
|
||||
fn play(&mut self, pcm: Vec<u8>);
|
||||
fn stop(&mut self);
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct PlaybackEnd {
|
||||
pub id: String,
|
||||
/// Sentences that were played.
|
||||
pub played: u32,
|
||||
pub cancelled: bool,
|
||||
pub failed: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum PlaybackEvent {
|
||||
Start(String),
|
||||
End(PlaybackEnd),
|
||||
}
|
||||
|
||||
struct Request {
|
||||
id: String,
|
||||
chunks: BTreeMap<u32, Vec<u8>>,
|
||||
next: u32,
|
||||
done: bool,
|
||||
failed: bool,
|
||||
played: u32,
|
||||
started: bool,
|
||||
}
|
||||
|
||||
pub struct Playback<B: TtsBackend, S: ClipSink> {
|
||||
backend: B,
|
||||
sink: S,
|
||||
queue: VecDeque<Request>,
|
||||
playing: bool,
|
||||
counter: u64,
|
||||
events: Vec<PlaybackEvent>,
|
||||
}
|
||||
|
||||
impl<B: TtsBackend, S: ClipSink> Playback<B, S> {
|
||||
pub fn new(backend: B, sink: S) -> Self {
|
||||
Self { backend, sink, queue: VecDeque::new(), playing: false, counter: 0, events: vec![] }
|
||||
}
|
||||
|
||||
/// True while a clip is playing or a request is waiting for its audio.
|
||||
pub fn busy(&self) -> bool {
|
||||
self.playing || !self.queue.is_empty()
|
||||
}
|
||||
|
||||
/// Start and end events since the last call.
|
||||
pub fn take_events(&mut self) -> Vec<PlaybackEvent> {
|
||||
std::mem::take(&mut self.events)
|
||||
}
|
||||
|
||||
/// Queue `text` as a new request; with `more`, add text later with `continue_text`. Returns its id.
|
||||
pub fn say(&mut self, text: &str, more: bool) -> String {
|
||||
self.counter += 1;
|
||||
let id = format!("r{}", self.counter);
|
||||
self.queue.push_back(Request { id: id.clone(), chunks: BTreeMap::new(), next: 0, done: false, failed: false,
|
||||
played: 0, started: false });
|
||||
crate::debug_log!("play", "say id={id} chars={} more={more} queued={}", text.chars().count(), self.queue.len());
|
||||
self.backend.tts(&id, text, more);
|
||||
id
|
||||
}
|
||||
|
||||
/// More text for an open request (an LLM answer that is still streaming).
|
||||
pub fn continue_text(&mut self, id: &str, text: &str, more: bool) {
|
||||
if self.queue.iter().any(|r| r.id == id && !r.done) {
|
||||
self.backend.tts(id, text, more);
|
||||
}
|
||||
}
|
||||
|
||||
/// A sentence of audio arrived from the worker.
|
||||
pub fn audio(&mut self, id: &str, seq: u32, pcm: Vec<u8>) {
|
||||
if let Some(r) = self.queue.iter_mut().find(|r| r.id == id) {
|
||||
r.chunks.insert(seq, pcm);
|
||||
self.pump();
|
||||
}
|
||||
}
|
||||
|
||||
/// The worker finished a request (all sentences sent, cancelled, or failed).
|
||||
pub fn done(&mut self, d: &TtsDone) {
|
||||
if let Some(r) = self.queue.iter_mut().find(|r| r.id == d.id) {
|
||||
r.done = true;
|
||||
r.failed = d.failed;
|
||||
self.pump();
|
||||
}
|
||||
}
|
||||
|
||||
/// The sink finished playing the current clip.
|
||||
pub fn clip_ended(&mut self) {
|
||||
self.playing = false;
|
||||
self.pump();
|
||||
}
|
||||
|
||||
/// Silence now: stop the clip, drop every queued request, cancel the worker's work.
|
||||
pub fn stop(&mut self) -> usize {
|
||||
let stopped: Vec<Request> = self.queue.drain(..).collect();
|
||||
let was_playing = self.playing;
|
||||
self.playing = false;
|
||||
for r in stopped.iter().filter(|r| !r.done) {
|
||||
self.backend.tts_cancel(&r.id);
|
||||
}
|
||||
if was_playing {
|
||||
self.sink.stop();
|
||||
}
|
||||
for r in &stopped {
|
||||
self.events.push(PlaybackEvent::End(PlaybackEnd { id: r.id.clone(), played: r.played, cancelled: true,
|
||||
failed: false }));
|
||||
}
|
||||
crate::debug_log!("play", "stop requests={} was_playing={was_playing}", stopped.len());
|
||||
stopped.len()
|
||||
}
|
||||
|
||||
fn pump(&mut self) {
|
||||
while !self.playing {
|
||||
let Some(head) = self.queue.front_mut() else { return };
|
||||
if !head.chunks.contains_key(&head.next) && head.done {
|
||||
// A sentence failed: skip its gap and play what did arrive.
|
||||
if let Some(&first) = head.chunks.keys().next() {
|
||||
head.next = first;
|
||||
}
|
||||
}
|
||||
if let Some(pcm) = head.chunks.remove(&head.next) {
|
||||
head.next += 1;
|
||||
head.played += 1;
|
||||
if !head.started {
|
||||
head.started = true;
|
||||
self.events.push(PlaybackEvent::Start(head.id.clone()));
|
||||
}
|
||||
self.playing = true;
|
||||
self.sink.play(pcm);
|
||||
return;
|
||||
}
|
||||
if !head.done {
|
||||
return;
|
||||
}
|
||||
let r = self.queue.pop_front().expect("head exists");
|
||||
crate::debug_log!("play", "said id={} sentences={} failed={}", r.id, r.played, r.failed);
|
||||
self.events.push(PlaybackEvent::End(PlaybackEnd { id: r.id, played: r.played, cancelled: false,
|
||||
failed: r.failed }));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::{cell::RefCell, rc::Rc};
|
||||
|
||||
#[derive(Default)]
|
||||
struct Log {
|
||||
sent: Vec<(String, String, bool)>,
|
||||
cancelled: Vec<String>,
|
||||
played: Vec<String>,
|
||||
stops: usize,
|
||||
}
|
||||
#[derive(Clone, Default)]
|
||||
struct Fake(Rc<RefCell<Log>>);
|
||||
impl TtsBackend for Fake {
|
||||
fn tts(&self, id: &str, text: &str, more: bool) -> bool {
|
||||
self.0.borrow_mut().sent.push((id.into(), text.into(), more));
|
||||
true
|
||||
}
|
||||
fn tts_cancel(&self, id: &str) {
|
||||
self.0.borrow_mut().cancelled.push(id.into());
|
||||
}
|
||||
}
|
||||
impl ClipSink for Fake {
|
||||
fn play(&mut self, pcm: Vec<u8>) {
|
||||
self.0.borrow_mut().played.push(String::from_utf8(pcm).unwrap());
|
||||
}
|
||||
fn stop(&mut self) {
|
||||
self.0.borrow_mut().stops += 1;
|
||||
}
|
||||
}
|
||||
|
||||
fn setup() -> (Playback<Fake, Fake>, Fake) {
|
||||
let f = Fake::default();
|
||||
(Playback::new(f.clone(), f.clone()), f)
|
||||
}
|
||||
fn done(id: &str, failed: bool) -> TtsDone {
|
||||
TtsDone { id: id.into(), chunks: 0, cancelled: false, failed }
|
||||
}
|
||||
fn end(id: &str, played: u32, cancelled: bool, failed: bool) -> PlaybackEvent {
|
||||
PlaybackEvent::End(PlaybackEnd { id: id.into(), played, cancelled, failed })
|
||||
}
|
||||
fn b(s: &str) -> Vec<u8> {
|
||||
s.as_bytes().to_vec()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plays_the_first_sentence_on_arrival_and_the_next_when_the_clip_ends() {
|
||||
let (mut p, f) = setup();
|
||||
let id = p.say("하나. 둘.", false);
|
||||
assert_eq!(f.0.borrow().sent, vec![(id.clone(), "하나. 둘.".into(), false)]);
|
||||
p.audio(&id, 0, b("s0"));
|
||||
assert_eq!(f.0.borrow().played, vec!["s0"]);
|
||||
p.audio(&id, 1, b("s1"));
|
||||
assert_eq!(f.0.borrow().played, vec!["s0"]);
|
||||
p.clip_ended();
|
||||
assert_eq!(f.0.borrow().played, vec!["s0", "s1"]);
|
||||
p.done(&done(&id, false));
|
||||
assert_eq!(p.take_events(), vec![PlaybackEvent::Start(id.clone())]);
|
||||
p.clip_ended();
|
||||
assert_eq!(p.take_events(), vec![end(&id, 2, false, false)]);
|
||||
assert!(!p.busy());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn requests_play_in_the_order_they_were_made() {
|
||||
let (mut p, f) = setup();
|
||||
let a = p.say("a", false);
|
||||
let c = p.say("b", false);
|
||||
p.audio(&c, 0, b("b0"));
|
||||
assert!(f.0.borrow().played.is_empty());
|
||||
p.audio(&a, 0, b("a0"));
|
||||
p.done(&done(&a, false));
|
||||
p.done(&done(&c, false));
|
||||
p.clip_ended();
|
||||
p.clip_ended();
|
||||
assert_eq!(f.0.borrow().played, vec!["a0", "b0"]);
|
||||
let ends: Vec<_> = p.take_events().into_iter().filter_map(|e| match e { PlaybackEvent::End(e) => Some(e.id), _ => None }).collect();
|
||||
assert_eq!(ends, vec![a, c]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sentences_stay_in_order_when_they_arrive_out_of_order() {
|
||||
let (mut p, f) = setup();
|
||||
let id = p.say("x", true);
|
||||
p.audio(&id, 1, b("s1"));
|
||||
assert!(f.0.borrow().played.is_empty());
|
||||
p.audio(&id, 0, b("s0"));
|
||||
p.clip_ended();
|
||||
assert_eq!(f.0.borrow().played, vec!["s0", "s1"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_failed_sentence_is_skipped_and_the_rest_still_play() {
|
||||
let (mut p, f) = setup();
|
||||
let id = p.say("a. b. c.", false);
|
||||
p.audio(&id, 0, b("s0"));
|
||||
p.audio(&id, 2, b("s2"));
|
||||
p.done(&done(&id, true));
|
||||
p.clip_ended();
|
||||
p.clip_ended();
|
||||
assert_eq!(f.0.borrow().played, vec!["s0", "s2"]);
|
||||
assert_eq!(p.take_events().last(), Some(&end(&id, 2, false, true)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stop_silences_cancels_unfinished_and_drops_the_queue() {
|
||||
let (mut p, f) = setup();
|
||||
let a = p.say("a", true);
|
||||
let c = p.say("b", false);
|
||||
p.audio(&a, 0, b("a0"));
|
||||
p.done(&done(&c, false));
|
||||
p.take_events();
|
||||
assert_eq!(p.stop(), 2);
|
||||
assert_eq!(f.0.borrow().stops, 1);
|
||||
assert_eq!(f.0.borrow().cancelled, vec![a.clone()]);
|
||||
assert_eq!(p.take_events(), vec![end(&a, 1, true, false), end(&c, 0, true, false)]);
|
||||
p.audio(&a, 1, b("a1"));
|
||||
p.clip_ended();
|
||||
assert_eq!(f.0.borrow().played, vec!["a0"]);
|
||||
assert!(!p.busy());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stop_with_nothing_to_say_touches_nothing() {
|
||||
let (mut p, f) = setup();
|
||||
assert_eq!(p.stop(), 0);
|
||||
assert_eq!(f.0.borrow().stops, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn text_can_be_added_to_an_open_request_but_not_a_finished_one() {
|
||||
let (mut p, f) = setup();
|
||||
let id = p.say("첫 문장.", true);
|
||||
p.continue_text(&id, " 둘째 문장.", false);
|
||||
p.done(&done(&id, false));
|
||||
p.continue_text(&id, "늦은 글", false);
|
||||
let sent: Vec<_> = f.0.borrow().sent.iter().map(|(_, t, m)| (t.clone(), *m)).collect();
|
||||
assert_eq!(sent, vec![("첫 문장.".into(), true), (" 둘째 문장.".into(), false)]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_request_the_worker_never_accepted_fails_without_blocking_the_next() {
|
||||
let (mut p, f) = setup();
|
||||
let a = p.say("a", false);
|
||||
let c = p.say("b", false);
|
||||
p.done(&done(&a, true));
|
||||
p.audio(&c, 0, b("b0"));
|
||||
assert_eq!(f.0.borrow().played, vec!["b0"]);
|
||||
assert_eq!(p.take_events()[0], end(&a, 0, false, true));
|
||||
}
|
||||
}
|
||||
271
app/src/receive.rs
Normal file
271
app/src/receive.rs
Normal file
@@ -0,0 +1,271 @@
|
||||
//! Per-speaker receive: RTP packets in arrival order -> Opus frames in sequence order -> PCM.
|
||||
//!
|
||||
//! songbird 0.6.1 decrypts (transport and DAVE) but feeds its own Opus decoder the payload without
|
||||
//! trimming what DAVE decryption left at the end, which garbles or drops about a third of the
|
||||
//! frames. The core therefore takes the decrypted packets with their exact payload bounds and
|
||||
//! decodes them itself. A small reorder window waits for at most `MAX_WAIT` later packets before a
|
||||
//! missing one is concealed (Opus in-band FEC from the next packet when it has it).
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
/// Later packets to wait for before giving up on a missing one: about 40 ms.
|
||||
pub const MAX_WAIT: usize = 2;
|
||||
/// A jump this far ahead (or any jump back past it) means the sender restarted its sequence.
|
||||
const RESYNC: i16 = 64;
|
||||
|
||||
pub trait FrameDecoder {
|
||||
/// Decode one Opus packet; `fec` decodes the previous (lost) frame from this packet's FEC data.
|
||||
fn decode(&mut self, packet: &[u8], fec: bool) -> Option<Vec<i16>>;
|
||||
/// Packet-loss concealment for one missing frame.
|
||||
fn conceal(&mut self) -> Option<Vec<i16>>;
|
||||
}
|
||||
|
||||
pub struct OpusFrames(opus2::Decoder);
|
||||
|
||||
impl OpusFrames {
|
||||
/// 48 kHz stereo, Discord's format.
|
||||
pub fn new() -> Self {
|
||||
Self(opus2::Decoder::new(48000, opus2::Channels::Stereo).expect("opus decoder"))
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for OpusFrames {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl FrameDecoder for OpusFrames {
|
||||
fn decode(&mut self, packet: &[u8], fec: bool) -> Option<Vec<i16>> {
|
||||
// FEC must ask for exactly the missing 20 ms; a normal decode gets room for 120 ms, the
|
||||
// largest Opus frame. Stereo.
|
||||
let mut out = vec![0i16; if fec { 960 * 2 } else { 5760 * 2 }];
|
||||
let n = self.0.decode(packet, &mut out, fec).ok()?;
|
||||
out.truncate(n * 2);
|
||||
Some(out)
|
||||
}
|
||||
fn conceal(&mut self) -> Option<Vec<i16>> {
|
||||
let mut out = vec![0i16; 960 * 2];
|
||||
let n = self.0.decode(&[], &mut out, false).ok()?;
|
||||
out.truncate(n * 2);
|
||||
Some(out)
|
||||
}
|
||||
}
|
||||
|
||||
/// One speaker's reorder window and decoder.
|
||||
pub struct SpeakerStream<D: FrameDecoder> {
|
||||
dec: D,
|
||||
next: Option<u16>,
|
||||
pending: BTreeMap<i16, Vec<u8>>,
|
||||
pub lost: u64,
|
||||
pub late: u64,
|
||||
pub bad: u64,
|
||||
}
|
||||
|
||||
impl<D: FrameDecoder> SpeakerStream<D> {
|
||||
pub fn new(dec: D) -> Self {
|
||||
Self { dec, next: None, pending: BTreeMap::new(), lost: 0, late: 0, bad: 0 }
|
||||
}
|
||||
|
||||
/// A packet arrived; returns the PCM frames now ready, in order.
|
||||
pub fn push(&mut self, seq: u16, opus: Vec<u8>) -> Vec<Vec<i16>> {
|
||||
let mut out = vec![];
|
||||
let next = *self.next.get_or_insert(seq);
|
||||
let ahead = seq.wrapping_sub(next) as i16;
|
||||
if !(-RESYNC..RESYNC).contains(&ahead) {
|
||||
// The sender restarted: drop the window and start over at this packet.
|
||||
self.flush_into(&mut out);
|
||||
self.next = Some(seq);
|
||||
self.decode_into(&opus, &mut out);
|
||||
self.next = Some(seq.wrapping_add(1));
|
||||
return out;
|
||||
}
|
||||
if ahead < 0 {
|
||||
self.late += 1;
|
||||
return out;
|
||||
}
|
||||
self.pending.insert(ahead, opus);
|
||||
self.drain(&mut out, false);
|
||||
out
|
||||
}
|
||||
|
||||
/// The speaker stopped: play out what is waiting, concealing nothing at the end.
|
||||
pub fn flush(&mut self) -> Vec<Vec<i16>> {
|
||||
let mut out = vec![];
|
||||
self.flush_into(&mut out);
|
||||
out
|
||||
}
|
||||
|
||||
fn flush_into(&mut self, out: &mut Vec<Vec<i16>>) {
|
||||
self.drain(out, true);
|
||||
}
|
||||
|
||||
fn decode_into(&mut self, opus: &[u8], out: &mut Vec<Vec<i16>>) {
|
||||
match self.dec.decode(opus, false) {
|
||||
Some(pcm) if !pcm.is_empty() => out.push(pcm),
|
||||
_ => self.bad += 1,
|
||||
}
|
||||
}
|
||||
|
||||
/// Play every packet that is due; when `force`, or the window holds more than MAX_WAIT packets
|
||||
/// past a gap, conceal the gap and go on.
|
||||
fn drain(&mut self, out: &mut Vec<Vec<i16>>, force: bool) {
|
||||
loop {
|
||||
if let Some(opus) = self.pending.remove(&0) {
|
||||
self.decode_into(&opus, out);
|
||||
self.advance(1);
|
||||
continue;
|
||||
}
|
||||
let Some(&first) = self.pending.keys().next() else { return };
|
||||
if !force && self.pending.len() <= MAX_WAIT {
|
||||
return;
|
||||
}
|
||||
// Frames 0..first are missing. Recover the last one from the next packet's FEC, conceal
|
||||
// the others.
|
||||
for k in 0..first {
|
||||
let pcm = if k == first - 1 {
|
||||
let next = self.pending.get(&first).cloned().unwrap_or_default();
|
||||
self.dec.decode(&next, true)
|
||||
} else {
|
||||
self.dec.conceal()
|
||||
};
|
||||
if let Some(pcm) = pcm.filter(|p| !p.is_empty()) {
|
||||
out.push(pcm);
|
||||
}
|
||||
self.lost += 1;
|
||||
}
|
||||
self.advance(first);
|
||||
}
|
||||
}
|
||||
|
||||
fn advance(&mut self, by: i16) {
|
||||
self.next = self.next.map(|n| n.wrapping_add(by as u16));
|
||||
self.pending = std::mem::take(&mut self.pending).into_iter().map(|(k, v)| (k - by, v)).collect();
|
||||
}
|
||||
}
|
||||
|
||||
/// The RTP sequence number and SSRC of a packet, and the Opus payload inside it given songbird's
|
||||
/// payload bounds (offset into the body after the fixed header and CSRCs, bytes to drop at the end).
|
||||
pub fn opus_payload(packet: &[u8], payload_offset: usize, payload_end_pad: usize) -> Option<(u16, u32, &[u8])> {
|
||||
if packet.len() < 12 {
|
||||
return None;
|
||||
}
|
||||
let csrc = usize::from(packet[0] & 0x0f);
|
||||
let header = 12 + 4 * csrc;
|
||||
let seq = u16::from_be_bytes([packet[2], packet[3]]);
|
||||
let ssrc = u32::from_be_bytes([packet[8], packet[9], packet[10], packet[11]]);
|
||||
let start = header.checked_add(payload_offset)?;
|
||||
let end = packet.len().checked_sub(payload_end_pad)?;
|
||||
(start < end).then(|| (seq, ssrc, &packet[start..end]))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// "Decodes" a packet to one sample holding its first byte; FEC and concealment are marked.
|
||||
struct Tag;
|
||||
impl FrameDecoder for Tag {
|
||||
fn decode(&mut self, p: &[u8], fec: bool) -> Option<Vec<i16>> {
|
||||
Some(vec![if fec { 1000 + p.first().copied().unwrap_or(0) as i16 } else { p[0] as i16 }])
|
||||
}
|
||||
fn conceal(&mut self) -> Option<Vec<i16>> {
|
||||
Some(vec![-1])
|
||||
}
|
||||
}
|
||||
|
||||
fn run(s: &mut SpeakerStream<Tag>, seqs: &[u16]) -> Vec<i16> {
|
||||
seqs.iter().flat_map(|&q| s.push(q, vec![q as u8])).map(|f| f[0]).collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn in_order_packets_play_at_once() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
assert_eq!(run(&mut s, &[5, 6, 7]), vec![5, 6, 7]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_swapped_pair_is_put_back_in_order() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
assert_eq!(run(&mut s, &[1, 3, 2, 4]), vec![1, 2, 3, 4]);
|
||||
assert_eq!((s.lost, s.late), (0, 0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_lost_packet_is_recovered_from_fec_after_the_wait() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
// 3 never comes; after MAX_WAIT + 1 later packets it is rebuilt from 4's FEC data.
|
||||
assert_eq!(run(&mut s, &[1, 2, 4, 5]), vec![1, 2]);
|
||||
assert_eq!(run(&mut s, &[6]), vec![1004, 4, 5, 6]);
|
||||
assert_eq!(s.lost, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn several_lost_packets_are_concealed_then_fec_for_the_last() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
assert_eq!(run(&mut s, &[1, 4, 5, 6]), vec![1, -1, 1004, 4, 5, 6]);
|
||||
assert_eq!(s.lost, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn late_and_duplicate_packets_are_dropped() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
assert_eq!(run(&mut s, &[1, 2, 3, 2, 1]), vec![1, 2, 3]);
|
||||
assert_eq!(s.late, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sequence_numbers_wrap() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
let out: Vec<i16> = [65534u16, 65535, 0, 1].iter().flat_map(|&q| s.push(q, vec![(q % 7) as u8])).map(|f| f[0]).collect();
|
||||
assert_eq!(out, vec![65534 % 7, 65535 % 7, 0, 1].into_iter().map(|v| v as i16).collect::<Vec<_>>());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_far_jump_starts_over() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
assert_eq!(run(&mut s, &[10, 11, 200, 201]), vec![10, 11, 200, 201]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flush_plays_what_waits_behind_a_gap() {
|
||||
let mut s = SpeakerStream::new(Tag);
|
||||
assert_eq!(run(&mut s, &[1, 3]), vec![1]);
|
||||
assert_eq!(s.flush().iter().map(|f| f[0]).collect::<Vec<_>>(), vec![1003, 3]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn real_opus_round_trip_through_the_window() {
|
||||
// Encode 5 frames of a tone, drop one, swap two: all 5 frames come out, in order.
|
||||
let mut enc = opus2::Encoder::new(48000, opus2::Channels::Stereo, opus2::Application::Voip).unwrap();
|
||||
enc.set_inband_fec(true).unwrap();
|
||||
enc.set_packet_loss_perc(10).unwrap();
|
||||
let frames: Vec<Vec<u8>> = (0..6).map(|k| {
|
||||
let pcm: Vec<i16> = (0..960).flat_map(|i| {
|
||||
let v = (8000.0 * (2.0 * std::f64::consts::PI * 440.0 * (k * 960 + i) as f64 / 48000.0).sin()) as i16;
|
||||
[v, v]
|
||||
}).collect();
|
||||
enc.encode_vec(&pcm, 4000).unwrap()
|
||||
}).collect();
|
||||
let mut s = SpeakerStream::new(OpusFrames::new());
|
||||
let mut out = vec![];
|
||||
for q in [0u16, 2, 1, 4, 5] {
|
||||
out.extend(s.push(q, frames[q as usize].clone()));
|
||||
}
|
||||
out.extend(s.flush());
|
||||
assert_eq!(out.len(), 6);
|
||||
assert!(out.iter().all(|f| f.len() == 1920));
|
||||
assert_eq!(s.lost, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_bounds_follow_the_header_and_songbirds_offsets() {
|
||||
let mut p = vec![0x80u8, 0x78, 0x01, 0x02, 0, 0, 0, 0, 0, 0, 0x30, 0x39];
|
||||
p.extend_from_slice(&[9, 9, 9, 9]); // extension (offset 4)
|
||||
p.extend_from_slice(&[1, 2, 3]); // opus
|
||||
p.extend_from_slice(&[7, 7]); // DAVE leftovers (end pad 2)
|
||||
let (seq, ssrc, opus) = opus_payload(&p, 4, 2).unwrap();
|
||||
assert_eq!((seq, ssrc, opus), (0x0102, 12345, &[1u8, 2, 3][..]));
|
||||
assert!(opus_payload(&p, 10, 9).is_none());
|
||||
}
|
||||
}
|
||||
65
app/src/services.rs
Normal file
65
app/src/services.rs
Normal file
@@ -0,0 +1,65 @@
|
||||
//! Starts everything the app runs: the voice worker process, the connection to it, and the Discord
|
||||
//! core. Used by the app binary (and the Tauri shell from M5).
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use serde_json::Value;
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
use crate::discord::{CoreEvent, DiscordCore};
|
||||
use crate::settings::load_settings;
|
||||
use crate::worker_client::{WorkerClient, WorkerClientOptions};
|
||||
use crate::worker_process::{ProcessEvent, WorkerProcess, WorkerProcessOptions};
|
||||
use crate::debug_log;
|
||||
|
||||
pub struct Services {
|
||||
pub core: DiscordCore,
|
||||
pub events: mpsc::UnboundedReceiver<CoreEvent>,
|
||||
pub worker: WorkerClient,
|
||||
pub worker_process: WorkerProcess,
|
||||
}
|
||||
|
||||
impl Services {
|
||||
pub async fn stop(&self) {
|
||||
self.core.close().await;
|
||||
self.worker.close();
|
||||
self.worker_process.stop().await;
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn start_services(data_dir: &Path) -> Result<Services, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let settings = load_settings(&data_dir.join("config").join("settings.json"))?;
|
||||
// Until the setup wizard stores it encrypted (M5), the token comes from the environment only.
|
||||
let token = std::env::var("JARVIS_DISCORD_TOKEN").map_err(|_| "JARVIS_DISCORD_TOKEN is not set")?;
|
||||
|
||||
let w = &settings.voice_worker;
|
||||
let worker_process = WorkerProcess::new(WorkerProcessOptions::new(&w.python, &w.cwd, w.args.clone()))?;
|
||||
let mut process_events = worker_process.subscribe();
|
||||
worker_process.start();
|
||||
let mut opts = WorkerClientOptions::new(worker_process.url(), worker_process.token.clone());
|
||||
opts.stt = settings.voice.stt.clone();
|
||||
opts.stt.insert("hotwords".into(), Value::from(settings.voice.wake_words.clone()));
|
||||
let (worker, worker_events) = WorkerClient::new(opts);
|
||||
// The worker loads its models before it listens: connect once it says READY (after restarts the
|
||||
// client keeps retrying on its own).
|
||||
let w2 = worker.clone();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
match process_events.recv().await {
|
||||
Ok(ProcessEvent::Ready) => return w2.start(),
|
||||
Ok(_) => {}
|
||||
Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => {}
|
||||
Err(_) => return,
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
let (core, events) = DiscordCore::new(settings, token, worker.clone(), worker_events);
|
||||
if let Err(e) = core.start().await {
|
||||
worker.close();
|
||||
worker_process.stop().await;
|
||||
return Err(Box::new(e));
|
||||
}
|
||||
debug_log!("app", "services started");
|
||||
Ok(Services { core, events, worker, worker_process })
|
||||
}
|
||||
@@ -1,57 +0,0 @@
|
||||
/**
|
||||
* Starts everything the app runs: the voice worker process, the connection to it, and the Discord
|
||||
* core. Shared by the Electron entry (main.ts) and the plain Node entry (cli.ts).
|
||||
*/
|
||||
import { join } from "node:path";
|
||||
import { DISCORD_PCM, DiscordCore } from "./core/discordCore.js";
|
||||
import { debugLog, errorLog } from "./core/log.js";
|
||||
import { loadSettings } from "./core/settings.js";
|
||||
import { VoiceWorkerClient } from "./core/workerClient.js";
|
||||
import { VoiceWorkerProcess } from "./core/workerProcess.js";
|
||||
|
||||
export interface Services {
|
||||
core: DiscordCore;
|
||||
worker: VoiceWorkerClient;
|
||||
workerProcess: VoiceWorkerProcess;
|
||||
stop(): Promise<void>;
|
||||
}
|
||||
|
||||
export async function startServices(dataDir: string): Promise<Services> {
|
||||
const settings = loadSettings(join(dataDir, "config", "settings.json"));
|
||||
// Until the setup wizard stores it encrypted (M5), the token comes from the environment only.
|
||||
const token = process.env.JARVIS_DISCORD_TOKEN;
|
||||
if (!token) throw new Error("JARVIS_DISCORD_TOKEN is not set");
|
||||
|
||||
const workerProcess = new VoiceWorkerProcess(settings.voiceWorker);
|
||||
await workerProcess.start();
|
||||
const worker = new VoiceWorkerClient({
|
||||
url: workerProcess.url,
|
||||
token: workerProcess.token,
|
||||
audio: DISCORD_PCM,
|
||||
tts: DISCORD_PCM,
|
||||
stt: { ...settings.voice.stt, hotwords: settings.voice.wakeWords },
|
||||
});
|
||||
// The worker loads its models before it listens; connect once it says READY (and after restarts
|
||||
// the client keeps retrying on its own).
|
||||
workerProcess.once("ready", () => worker.start());
|
||||
|
||||
const core = new DiscordCore(settings, token, worker);
|
||||
core.on("final", (f) => debugLog(`heard ${f.speakerId} utt=${f.utt} chars=${f.text.length}`, "app"));
|
||||
try {
|
||||
await core.start();
|
||||
} catch (e) {
|
||||
worker.close();
|
||||
await workerProcess.stop();
|
||||
throw e;
|
||||
}
|
||||
return {
|
||||
core,
|
||||
worker,
|
||||
workerProcess,
|
||||
async stop() {
|
||||
await core.close().catch((e) => errorLog(`discord close: ${(e as Error).message}`));
|
||||
worker.close();
|
||||
await workerProcess.stop();
|
||||
},
|
||||
};
|
||||
}
|
||||
353
app/src/settings.rs
Normal file
353
app/src/settings.rs
Normal file
@@ -0,0 +1,353 @@
|
||||
//! App settings: `<data folder>/config/settings.json` (see `app/src/discord_core.spec.md`).
|
||||
//!
|
||||
//! Every key is optional in the file; missing keys take the defaults. A bad value stops the app with
|
||||
//! a message naming the key, rather than running with a setting the user did not choose. Unknown keys
|
||||
//! are ignored.
|
||||
|
||||
use serde_json::{Map, Value};
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ChatChannelSettings {
|
||||
pub id: u64,
|
||||
/// What the channel is for, as the user describes it; given to the assistant (M3).
|
||||
pub name: String,
|
||||
pub description: String,
|
||||
/// The assistant may read this channel's messages.
|
||||
pub read: bool,
|
||||
/// The assistant may send messages to this channel.
|
||||
pub send: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct DiscordSettings {
|
||||
/// The one server the bot works in. The slash command is registered there.
|
||||
pub guild_id: Option<u64>,
|
||||
/// Voice channel to join on start ("자동 입장").
|
||||
pub voice_channel_id: Option<u64>,
|
||||
pub command_name: String,
|
||||
/// Slash command replies only the caller sees.
|
||||
pub ephemeral_replies: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct VoiceSettings {
|
||||
/// Most people transcribed at once (1-10).
|
||||
pub max_speakers: usize,
|
||||
/// Transcribe other bots too (music bots are ignored by default).
|
||||
pub listen_to_bots: bool,
|
||||
/// Posted in the voice channel's chat on every join (privacy.md).
|
||||
pub join_notice: String,
|
||||
/// Wake words; passed to the STT as a hint now, used for call detection from M4.
|
||||
pub wake_words: Vec<String>,
|
||||
/// Voice worker STT settings (voice_worker.spec.md); the worker validates them.
|
||||
pub stt: Map<String, Value>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct WorkerSettings {
|
||||
/// Python of the voice worker venv.
|
||||
pub python: String,
|
||||
/// Directory holding the `voice_worker` package (the repo's `src/`).
|
||||
pub cwd: PathBuf,
|
||||
/// Extra worker arguments, e.g. `["--melo-python", "<MeloTTS venv python>"]`.
|
||||
pub args: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct Settings {
|
||||
pub discord: DiscordSettings,
|
||||
pub voice: VoiceSettings,
|
||||
pub chat_channels: Vec<ChatChannelSettings>,
|
||||
pub voice_worker: WorkerSettings,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct SettingsError(pub String);
|
||||
|
||||
impl std::fmt::Display for SettingsError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str(&self.0)
|
||||
}
|
||||
}
|
||||
impl std::error::Error for SettingsError {}
|
||||
|
||||
pub const DEFAULT_JOIN_NOTICE: &str =
|
||||
"자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다.";
|
||||
|
||||
impl Default for Settings {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
discord: DiscordSettings {
|
||||
guild_id: None,
|
||||
voice_channel_id: None,
|
||||
command_name: "자비스".into(),
|
||||
ephemeral_replies: true,
|
||||
},
|
||||
voice: VoiceSettings {
|
||||
max_speakers: 10,
|
||||
listen_to_bots: false,
|
||||
join_notice: DEFAULT_JOIN_NOTICE.into(),
|
||||
wake_words: vec!["자비스".into()],
|
||||
stt: Map::new(),
|
||||
},
|
||||
chat_channels: vec![],
|
||||
voice_worker: WorkerSettings {
|
||||
python: if cfg!(windows) { "python.exe".into() } else { "python3".into() },
|
||||
// The repo's src/, next to this crate.
|
||||
cwd: Path::new(env!("CARGO_MANIFEST_DIR")).join("..").join("src"),
|
||||
args: vec![],
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn err<T>(msg: impl Into<String>) -> Result<T, SettingsError> {
|
||||
Err(SettingsError(msg.into()))
|
||||
}
|
||||
|
||||
fn section<'a>(raw: &'a Map<String, Value>, key: &str) -> Result<Option<&'a Map<String, Value>>, SettingsError> {
|
||||
match raw.get(key) {
|
||||
None => Ok(None),
|
||||
Some(Value::Object(m)) => Ok(Some(m)),
|
||||
Some(_) => err(format!("{key} must be an object")),
|
||||
}
|
||||
}
|
||||
|
||||
/// A Discord ID: a JSON string of 17-20 digits (numbers would lose precision in JSON).
|
||||
fn snowflake(v: &Value) -> Option<u64> {
|
||||
let s = v.as_str()?;
|
||||
if !(17..=20).contains(&s.len()) || !s.bytes().all(|b| b.is_ascii_digit()) {
|
||||
return None;
|
||||
}
|
||||
s.parse().ok()
|
||||
}
|
||||
|
||||
fn opt_snowflake(m: Option<&Map<String, Value>>, key: &str, path: &str, default: Option<u64>, what: &str)
|
||||
-> Result<Option<u64>, SettingsError> {
|
||||
match m.and_then(|m| m.get(key)) {
|
||||
None => Ok(default),
|
||||
Some(Value::Null) => Ok(None),
|
||||
Some(v) => snowflake(v).map(Some).ok_or_else(|| SettingsError(format!("{path}.{key} must be {what} or null"))),
|
||||
}
|
||||
}
|
||||
|
||||
fn get_str(m: Option<&Map<String, Value>>, key: &str, path: &str, default: &str) -> Result<String, SettingsError> {
|
||||
match m.and_then(|m| m.get(key)) {
|
||||
None => Ok(default.to_owned()),
|
||||
Some(Value::String(s)) => Ok(s.clone()),
|
||||
Some(_) => err(format!("{path}.{key} must be a string")),
|
||||
}
|
||||
}
|
||||
|
||||
fn get_bool(m: Option<&Map<String, Value>>, key: &str, path: &str, default: bool) -> Result<bool, SettingsError> {
|
||||
match m.and_then(|m| m.get(key)) {
|
||||
None => Ok(default),
|
||||
Some(Value::Bool(b)) => Ok(*b),
|
||||
Some(_) => err(format!("{path}.{key} must be true or false")),
|
||||
}
|
||||
}
|
||||
|
||||
fn get_str_list(m: Option<&Map<String, Value>>, key: &str, path: &str, default: &[String])
|
||||
-> Result<Vec<String>, SettingsError> {
|
||||
match m.and_then(|m| m.get(key)) {
|
||||
None => Ok(default.to_vec()),
|
||||
Some(Value::Array(a)) => a
|
||||
.iter()
|
||||
.map(|v| v.as_str().map(str::to_owned))
|
||||
.collect::<Option<Vec<_>>>()
|
||||
.ok_or_else(|| SettingsError(format!("{path}.{key} must be a list of strings"))),
|
||||
Some(_) => err(format!("{path}.{key} must be a list of strings")),
|
||||
}
|
||||
}
|
||||
|
||||
/// Discord's rule for command names: 1-32 lowercase letters, digits, `-` or `_`.
|
||||
fn valid_command_name(s: &str) -> bool {
|
||||
let n = s.chars().count();
|
||||
(1..=32).contains(&n)
|
||||
&& s.chars().all(|c| c == '-' || c == '_' || c.is_alphanumeric())
|
||||
&& s.to_lowercase() == s
|
||||
}
|
||||
|
||||
/// Validate a parsed settings document over the defaults.
|
||||
pub fn parse_settings(raw: &Value) -> Result<Settings, SettingsError> {
|
||||
let d = Settings::default();
|
||||
let Value::Object(raw) = raw else { return err("settings must be a JSON object") };
|
||||
let disc = section(raw, "discord")?;
|
||||
let voice = section(raw, "voice")?;
|
||||
let worker = section(raw, "voiceWorker")?;
|
||||
|
||||
let command_name = get_str(disc, "commandName", "discord", &d.discord.command_name)?;
|
||||
if !valid_command_name(&command_name) {
|
||||
return err("discord.commandName must be 1-32 lowercase letters, digits, - or _");
|
||||
}
|
||||
let max_speakers = match voice.and_then(|m| m.get("maxSpeakers")) {
|
||||
None => d.voice.max_speakers,
|
||||
Some(v) => match v.as_u64() {
|
||||
Some(n @ 1..=10) => n as usize,
|
||||
_ => return err("voice.maxSpeakers must be an integer 1-10"),
|
||||
},
|
||||
};
|
||||
let wake_words: Vec<String> = get_str_list(voice, "wakeWords", "voice", &d.voice.wake_words)?
|
||||
.into_iter()
|
||||
.map(|w| w.trim().to_owned())
|
||||
.filter(|w| !w.is_empty())
|
||||
.collect();
|
||||
let join_notice = get_str(voice, "joinNotice", "voice", &d.voice.join_notice)?.trim().to_owned();
|
||||
if join_notice.is_empty() {
|
||||
return err("voice.joinNotice must not be empty: people must be told they are transcribed");
|
||||
}
|
||||
if join_notice.encode_utf16().count() > 2000 {
|
||||
return err("voice.joinNotice must be at most 2000 characters");
|
||||
}
|
||||
let stt = match voice.and_then(|m| m.get("stt")) {
|
||||
None => d.voice.stt.clone(),
|
||||
Some(Value::Object(m)) => m.clone(),
|
||||
Some(_) => return err("voice.stt must be an object"),
|
||||
};
|
||||
|
||||
let chat_raw = match raw.get("chatChannels") {
|
||||
None => vec![],
|
||||
Some(Value::Array(a)) => a.clone(),
|
||||
Some(_) => return err("chatChannels must be a list"),
|
||||
};
|
||||
let mut chat_channels: Vec<ChatChannelSettings> = Vec::new();
|
||||
for (i, c) in chat_raw.iter().enumerate() {
|
||||
let path = format!("chatChannels[{i}]");
|
||||
let Value::Object(m) = c else { return err(format!("{path} must be an object")) };
|
||||
let id = m.get("id").and_then(snowflake).ok_or_else(|| SettingsError(format!("{path}.id must be a Discord channel ID")))?;
|
||||
if chat_channels.iter().any(|c| c.id == id) {
|
||||
return err(format!("{path}.id is listed twice"));
|
||||
}
|
||||
chat_channels.push(ChatChannelSettings {
|
||||
id,
|
||||
name: get_str(Some(m), "name", &path, "")?,
|
||||
description: get_str(Some(m), "description", &path, "")?,
|
||||
read: get_bool(Some(m), "read", &path, true)?,
|
||||
send: get_bool(Some(m), "send", &path, true)?,
|
||||
});
|
||||
}
|
||||
|
||||
Ok(Settings {
|
||||
discord: DiscordSettings {
|
||||
guild_id: opt_snowflake(disc, "guildId", "discord", d.discord.guild_id, "a Discord server ID")?,
|
||||
voice_channel_id: opt_snowflake(disc, "voiceChannelId", "discord", d.discord.voice_channel_id,
|
||||
"a Discord channel ID")?,
|
||||
command_name,
|
||||
ephemeral_replies: get_bool(disc, "ephemeralReplies", "discord", d.discord.ephemeral_replies)?,
|
||||
},
|
||||
voice: VoiceSettings {
|
||||
max_speakers,
|
||||
listen_to_bots: get_bool(voice, "listenToBots", "voice", d.voice.listen_to_bots)?,
|
||||
join_notice,
|
||||
wake_words,
|
||||
stt,
|
||||
},
|
||||
chat_channels,
|
||||
voice_worker: WorkerSettings {
|
||||
python: get_str(worker, "python", "voiceWorker", &d.voice_worker.python)?,
|
||||
cwd: match worker.and_then(|m| m.get("cwd")) {
|
||||
None => d.voice_worker.cwd.clone(),
|
||||
Some(Value::String(s)) => PathBuf::from(s),
|
||||
Some(_) => return err("voiceWorker.cwd must be a string"),
|
||||
},
|
||||
args: get_str_list(worker, "args", "voiceWorker", &d.voice_worker.args)?,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
/// Read and validate the settings file. A missing file means all defaults.
|
||||
pub fn load_settings(path: &Path) -> Result<Settings, SettingsError> {
|
||||
let text = match std::fs::read_to_string(path) {
|
||||
Ok(t) => t,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Settings::default()),
|
||||
Err(e) => return err(format!("could not read {}: {e}", path.display())),
|
||||
};
|
||||
let raw: Value = serde_json::from_str(text.trim_start_matches('\u{feff}'))
|
||||
.map_err(|e| SettingsError(format!("{} is not valid JSON: {e}", path.display())))?;
|
||||
parse_settings(&raw)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
const ID: &str = "123456789012345678";
|
||||
const IDN: u64 = 123456789012345678;
|
||||
|
||||
#[test]
|
||||
fn empty_document_and_missing_file_give_the_defaults() {
|
||||
assert_eq!(parse_settings(&json!({})).unwrap(), Settings::default());
|
||||
let dir = std::env::temp_dir().join(format!("jarvis-settings-{}", std::process::id()));
|
||||
assert_eq!(load_settings(&dir.join("nope.json")).unwrap(), Settings::default());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reads_a_full_file_with_bom() {
|
||||
let dir = std::env::temp_dir().join(format!("jarvis-settings-full-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let path = dir.join("settings.json");
|
||||
let doc = json!({
|
||||
"discord": {"guildId": ID, "voiceChannelId": ID, "commandName": "jarvis", "ephemeralReplies": false},
|
||||
"voice": {"maxSpeakers": 3, "listenToBots": true, "joinNotice": "알림", "wakeWords": [" 자비스 ", "", "javis"],
|
||||
"stt": {"end_silence_ms": 250}},
|
||||
"chatChannels": [{"id": ID, "name": "일정", "description": "일정 올리는 곳", "send": false}],
|
||||
"voiceWorker": {"python": "/venv/bin/python", "cwd": "/repo/src", "args": ["--melo-python", "/m/python"]},
|
||||
"somethingNew": 1
|
||||
});
|
||||
std::fs::write(&path, format!("\u{feff}{doc}")).unwrap();
|
||||
let s = load_settings(&path).unwrap();
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
assert_eq!(s.discord, DiscordSettings { guild_id: Some(IDN), voice_channel_id: Some(IDN),
|
||||
command_name: "jarvis".into(), ephemeral_replies: false });
|
||||
assert_eq!(s.voice.max_speakers, 3);
|
||||
assert!(s.voice.listen_to_bots);
|
||||
assert_eq!(s.voice.join_notice, "알림");
|
||||
assert_eq!(s.voice.wake_words, vec!["자비스", "javis"]);
|
||||
assert_eq!(s.voice.stt.get("end_silence_ms"), Some(&json!(250)));
|
||||
assert_eq!(s.chat_channels, vec![ChatChannelSettings { id: IDN, name: "일정".into(),
|
||||
description: "일정 올리는 곳".into(), read: true, send: false }]);
|
||||
assert_eq!(s.voice_worker.args, vec!["--melo-python", "/m/python"]);
|
||||
assert_eq!(s.voice_worker.cwd, PathBuf::from("/repo/src"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bad_values_are_refused_with_the_key() {
|
||||
let cases = [
|
||||
(json!({"discord": {"guildId": "abc"}}), "discord.guildId"),
|
||||
(json!({"discord": {"guildId": 123456789012345678u64}}), "discord.guildId"),
|
||||
(json!({"discord": {"commandName": "Jarvis"}}), "commandName"),
|
||||
(json!({"discord": {"commandName": "두 단어"}}), "commandName"),
|
||||
(json!({"voice": {"maxSpeakers": 11}}), "maxSpeakers"),
|
||||
(json!({"voice": {"maxSpeakers": 0}}), "maxSpeakers"),
|
||||
(json!({"voice": {"joinNotice": " "}}), "joinNotice"),
|
||||
(json!({"chatChannels": [{"id": ID}, {"id": ID}]}), "twice"),
|
||||
(json!({"chatChannels": [{"id": "1"}]}), "chatChannels[0].id"),
|
||||
(json!({"chatChannels": {}}), "chatChannels"),
|
||||
(json!({"voice": []}), "voice"),
|
||||
(json!([]), "object"),
|
||||
];
|
||||
for (raw, key) in cases {
|
||||
let e = parse_settings(&raw).unwrap_err();
|
||||
assert!(e.0.contains(key), "{raw} -> {e}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn null_ids_clear_them() {
|
||||
let s = parse_settings(&json!({"discord": {"guildId": null, "voiceChannelId": null}})).unwrap();
|
||||
assert_eq!((s.discord.guild_id, s.discord.voice_channel_id), (None, None));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_json_names_the_file() {
|
||||
let dir = std::env::temp_dir().join(format!("jarvis-settings-bad-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let path = dir.join("settings.json");
|
||||
std::fs::write(&path, "{ nope").unwrap();
|
||||
let e = load_settings(&path).unwrap_err();
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
assert!(e.0.contains("not valid JSON"));
|
||||
}
|
||||
}
|
||||
70
app/src/speakers.rs
Normal file
70
app/src/speakers.rs
Normal file
@@ -0,0 +1,70 @@
|
||||
//! Who is being transcribed right now. At most `max` people at once (settings `voice.maxSpeakers`);
|
||||
//! someone who starts while the limit is reached is not heard until their next start after the
|
||||
//! others free a place.
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
pub struct SpeakerGate {
|
||||
max: usize,
|
||||
active: HashSet<u64>,
|
||||
}
|
||||
|
||||
impl SpeakerGate {
|
||||
pub fn new(max: usize) -> Self {
|
||||
Self { max, active: HashSet::new() }
|
||||
}
|
||||
|
||||
/// `id` started speaking. True if their audio goes to the STT.
|
||||
pub fn start(&mut self, id: u64) -> bool {
|
||||
if self.active.contains(&id) {
|
||||
return true;
|
||||
}
|
||||
if self.active.len() >= self.max {
|
||||
return false;
|
||||
}
|
||||
self.active.insert(id);
|
||||
true
|
||||
}
|
||||
|
||||
/// `id` stopped (or left). True if they were being transcribed.
|
||||
pub fn end(&mut self, id: u64) -> bool {
|
||||
self.active.remove(&id)
|
||||
}
|
||||
|
||||
pub fn is_active(&self, id: u64) -> bool {
|
||||
self.active.contains(&id)
|
||||
}
|
||||
|
||||
pub fn count(&self) -> usize {
|
||||
self.active.len()
|
||||
}
|
||||
|
||||
pub fn clear(&mut self) {
|
||||
self.active.clear();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn hears_up_to_the_limit_and_lets_the_next_in_when_someone_stops() {
|
||||
let mut g = SpeakerGate::new(2);
|
||||
assert!(g.start(1) && g.start(2));
|
||||
assert!(!g.start(3));
|
||||
assert!(!g.is_active(3));
|
||||
assert!(g.end(1));
|
||||
assert!(g.start(3));
|
||||
assert_eq!(g.count(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn already_heard_stays_heard_and_ending_a_stranger_is_a_no_op() {
|
||||
let mut g = SpeakerGate::new(1);
|
||||
assert!(g.start(1) && g.start(1));
|
||||
assert!(!g.end(9));
|
||||
assert!(g.end(1));
|
||||
assert!(!g.end(1));
|
||||
}
|
||||
}
|
||||
401
app/src/worker_client.rs
Normal file
401
app/src/worker_client.rs
Normal file
@@ -0,0 +1,401 @@
|
||||
//! WebSocket client for the Python voice worker (`src/voice_worker/voice_worker.spec.md`).
|
||||
//!
|
||||
//! Keeps one connection open, says hello with the app's token, and reconnects with backoff when the
|
||||
//! worker restarts. While disconnected, audio is dropped (the worker would not know the speaker's
|
||||
//! state anyway) and every open TTS request ends as failed, so playback never waits forever.
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use futures_util::{SinkExt, StreamExt};
|
||||
use serde::Deserialize;
|
||||
use serde_json::{json, Map, Value};
|
||||
use tokio::sync::{mpsc, watch, Notify};
|
||||
use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;
|
||||
use tokio_tungstenite::tungstenite::Message;
|
||||
|
||||
use crate::frames::{decode_tts_frame, encode_audio_frame, KIND_TTS};
|
||||
use crate::playback::TtsBackend;
|
||||
use crate::{debug_log, error_log};
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct PcmFormat {
|
||||
pub rate: u32,
|
||||
pub channels: u8,
|
||||
}
|
||||
|
||||
/// Discord voice after Opus decoding, and what the worker is asked to send back.
|
||||
pub const DISCORD_PCM: PcmFormat = PcmFormat { rate: 48000, channels: 2 };
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct WorkerClientOptions {
|
||||
pub url: String,
|
||||
pub token: String,
|
||||
pub audio: PcmFormat,
|
||||
pub tts: PcmFormat,
|
||||
pub stt: Map<String, Value>,
|
||||
/// First reconnect delay; doubles up to `max_backoff`.
|
||||
pub backoff: Duration,
|
||||
pub max_backoff: Duration,
|
||||
pub hello_timeout: Duration,
|
||||
}
|
||||
|
||||
impl WorkerClientOptions {
|
||||
pub fn new(url: impl Into<String>, token: impl Into<String>) -> Self {
|
||||
Self {
|
||||
url: url.into(),
|
||||
token: token.into(),
|
||||
audio: DISCORD_PCM,
|
||||
tts: DISCORD_PCM,
|
||||
stt: Map::new(),
|
||||
backoff: Duration::from_millis(500),
|
||||
max_backoff: Duration::from_secs(5),
|
||||
hello_timeout: Duration::from_secs(5),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
pub struct HelloOk {
|
||||
pub stt_ready: bool,
|
||||
pub tts_ready: bool,
|
||||
#[serde(default)]
|
||||
pub stt: Map<String, Value>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
|
||||
pub struct SpeechStart {
|
||||
pub speaker: String,
|
||||
pub utt: u32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
|
||||
pub struct Partial {
|
||||
pub speaker: String,
|
||||
pub utt: u32,
|
||||
pub text: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
|
||||
pub struct Final {
|
||||
pub speaker: String,
|
||||
pub utt: u32,
|
||||
pub text: String,
|
||||
pub end_reason: String,
|
||||
pub audio_ms: u64,
|
||||
pub wait_ms: u64,
|
||||
pub stt_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
|
||||
pub struct TtsDone {
|
||||
pub id: String,
|
||||
pub chunks: u32,
|
||||
pub cancelled: bool,
|
||||
pub failed: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
|
||||
pub struct WorkerError {
|
||||
pub code: String,
|
||||
pub message: String,
|
||||
#[serde(default)]
|
||||
pub id: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum WorkerEvent {
|
||||
Ready(HelloOk),
|
||||
Disconnected,
|
||||
SpeechStart(SpeechStart),
|
||||
Partial(Partial),
|
||||
Final(Final),
|
||||
TtsAudio { id: String, seq: u32, pcm: Vec<u8> },
|
||||
TtsDone(TtsDone),
|
||||
Error(WorkerError),
|
||||
}
|
||||
|
||||
struct State {
|
||||
conn: Option<mpsc::UnboundedSender<Message>>,
|
||||
stt: Map<String, Value>,
|
||||
/// TTS requests sent and not yet done, so a lost connection can end them.
|
||||
open_tts: HashSet<String>,
|
||||
closed: bool,
|
||||
dropped_frames: u64,
|
||||
}
|
||||
|
||||
struct Inner {
|
||||
opts: WorkerClientOptions,
|
||||
state: Mutex<State>,
|
||||
events: mpsc::UnboundedSender<WorkerEvent>,
|
||||
ready: watch::Sender<Option<HelloOk>>,
|
||||
wake: Notify,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct WorkerClient {
|
||||
inner: Arc<Inner>,
|
||||
}
|
||||
|
||||
impl WorkerClient {
|
||||
/// The client and the stream of worker events. Call `start()` to connect.
|
||||
pub fn new(opts: WorkerClientOptions) -> (Self, mpsc::UnboundedReceiver<WorkerEvent>) {
|
||||
let (events, rx) = mpsc::unbounded_channel();
|
||||
let stt = opts.stt.clone();
|
||||
let inner = Arc::new(Inner {
|
||||
opts,
|
||||
state: Mutex::new(State { conn: None, stt, open_tts: HashSet::new(), closed: false, dropped_frames: 0 }),
|
||||
events,
|
||||
ready: watch::channel(None).0,
|
||||
wake: Notify::new(),
|
||||
});
|
||||
(Self { inner }, rx)
|
||||
}
|
||||
|
||||
/// Start connecting; reconnects until `close()`.
|
||||
pub fn start(&self) {
|
||||
let inner = self.inner.clone();
|
||||
tokio::spawn(async move { run(inner).await });
|
||||
}
|
||||
|
||||
pub fn is_ready(&self) -> bool {
|
||||
self.inner.ready.borrow().is_some()
|
||||
}
|
||||
|
||||
/// Resolve once the worker has said hello_ok, or None after `timeout`.
|
||||
pub async fn wait_ready(&self, timeout: Duration) -> Option<HelloOk> {
|
||||
let mut rx = self.inner.ready.subscribe();
|
||||
tokio::time::timeout(timeout, async {
|
||||
loop {
|
||||
if let Some(h) = rx.borrow_and_update().clone() {
|
||||
return h;
|
||||
}
|
||||
if rx.changed().await.is_err() {
|
||||
std::future::pending::<()>().await;
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.ok()
|
||||
}
|
||||
|
||||
pub fn close(&self) {
|
||||
{
|
||||
let mut st = self.inner.state.lock().unwrap();
|
||||
st.closed = true;
|
||||
if let Some(tx) = st.conn.take() {
|
||||
let _ = tx.send(Message::Close(None));
|
||||
}
|
||||
}
|
||||
self.inner.wake.notify_waiters();
|
||||
self.inner.wake.notify_one();
|
||||
mark_disconnected(&self.inner);
|
||||
}
|
||||
|
||||
fn send(&self, msg: Message) -> bool {
|
||||
let st = self.inner.state.lock().unwrap();
|
||||
match &st.conn {
|
||||
Some(tx) if self.inner.ready.borrow().is_some() => tx.send(msg).is_ok(),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
fn send_json(&self, v: Value) -> bool {
|
||||
self.send(Message::text(v.to_string()))
|
||||
}
|
||||
|
||||
/// One frame of a speaker's PCM in the hello "audio" format. Dropped while disconnected.
|
||||
pub fn send_audio(&self, speaker: &str, pcm: &[i16]) {
|
||||
let Some(frame) = encode_audio_frame(speaker, pcm) else { return };
|
||||
if !self.send(Message::binary(frame)) {
|
||||
self.inner.state.lock().unwrap().dropped_frames += 1;
|
||||
}
|
||||
}
|
||||
|
||||
pub fn speaking_end(&self, speaker: &str) {
|
||||
self.send_json(json!({"type": "speaking_end", "speaker": speaker}));
|
||||
}
|
||||
|
||||
pub fn speaker_leave(&self, speaker: &str) {
|
||||
self.send_json(json!({"type": "speaker_leave", "speaker": speaker}));
|
||||
}
|
||||
|
||||
/// Change STT settings now and for every later connection.
|
||||
pub fn configure(&self, changes: Map<String, Value>) {
|
||||
self.inner.state.lock().unwrap().stt.extend(changes.clone());
|
||||
let mut msg = changes;
|
||||
msg.insert("type".into(), "config".into());
|
||||
self.send_json(Value::Object(msg));
|
||||
}
|
||||
}
|
||||
|
||||
impl TtsBackend for WorkerClient {
|
||||
/// Speak `text` as request `id`; with `more` the request stays open for more text. Returns false
|
||||
/// (and the request ends as failed) when the worker is not connected.
|
||||
fn tts(&self, id: &str, text: &str, more: bool) -> bool {
|
||||
let fresh = self.inner.state.lock().unwrap().open_tts.insert(id.to_owned());
|
||||
let sent = self.send_json(json!({"type": "tts", "id": id, "text": text, "more": more}));
|
||||
if !sent {
|
||||
let was_open = self.inner.state.lock().unwrap().open_tts.remove(id);
|
||||
if fresh || was_open {
|
||||
let _ = self.inner.events.send(WorkerEvent::TtsDone(TtsDone { id: id.into(), chunks: 0,
|
||||
cancelled: false, failed: true }));
|
||||
}
|
||||
}
|
||||
sent
|
||||
}
|
||||
|
||||
fn tts_cancel(&self, id: &str) {
|
||||
self.send_json(json!({"type": "tts_cancel", "id": id}));
|
||||
}
|
||||
}
|
||||
|
||||
fn mark_disconnected(inner: &Inner) {
|
||||
let was = inner.ready.send_replace(None).is_some();
|
||||
let (open, dropped) = {
|
||||
let mut st = inner.state.lock().unwrap();
|
||||
st.conn = None;
|
||||
(std::mem::take(&mut st.open_tts), st.dropped_frames)
|
||||
};
|
||||
for id in open {
|
||||
let _ = inner.events.send(WorkerEvent::TtsDone(TtsDone { id, chunks: 0, cancelled: false, failed: true }));
|
||||
}
|
||||
if was {
|
||||
debug_log!("worker", "voice worker disconnected (dropped audio frames so far: {dropped})");
|
||||
let _ = inner.events.send(WorkerEvent::Disconnected);
|
||||
}
|
||||
}
|
||||
|
||||
async fn run(inner: Arc<Inner>) {
|
||||
let mut backoff = inner.opts.backoff;
|
||||
loop {
|
||||
if inner.state.lock().unwrap().closed {
|
||||
return;
|
||||
}
|
||||
if connect_once(&inner).await {
|
||||
backoff = inner.opts.backoff;
|
||||
}
|
||||
mark_disconnected(&inner);
|
||||
if inner.state.lock().unwrap().closed {
|
||||
return;
|
||||
}
|
||||
tokio::select! {
|
||||
_ = tokio::time::sleep(backoff) => {}
|
||||
_ = inner.wake.notified() => {}
|
||||
}
|
||||
backoff = (backoff * 2).min(inner.opts.max_backoff);
|
||||
}
|
||||
}
|
||||
|
||||
/// One connection's life. True if it got as far as hello_ok.
|
||||
async fn connect_once(inner: &Arc<Inner>) -> bool {
|
||||
let ws = match tokio_tungstenite::connect_async(inner.opts.url.as_str()).await {
|
||||
Ok((ws, _)) => ws,
|
||||
Err(e) => {
|
||||
// Refused is normal while the worker is still loading its models.
|
||||
debug_log!("worker", "voice worker connect failed: {e}");
|
||||
return false;
|
||||
}
|
||||
};
|
||||
let (mut write, mut read) = ws.split();
|
||||
let stt = inner.state.lock().unwrap().stt.clone();
|
||||
let o = &inner.opts;
|
||||
let hello = json!({"type": "hello", "token": o.token,
|
||||
"audio": {"rate": o.audio.rate, "channels": o.audio.channels},
|
||||
"tts": {"rate": o.tts.rate, "channels": o.tts.channels}, "stt": stt});
|
||||
if write.send(Message::text(hello.to_string())).await.is_err() {
|
||||
return false;
|
||||
}
|
||||
let hello_ok = tokio::time::timeout(o.hello_timeout, async {
|
||||
while let Some(Ok(msg)) = read.next().await {
|
||||
match msg {
|
||||
Message::Text(t) => {
|
||||
let v: Value = serde_json::from_str(t.as_str()).ok()?;
|
||||
if v["type"] == "hello_ok" {
|
||||
return serde_json::from_value::<HelloOk>(v).ok();
|
||||
}
|
||||
}
|
||||
Message::Close(Some(f)) => {
|
||||
if matches!(f.code, CloseCode::Library(4400) | CloseCode::Library(4401)) {
|
||||
error_log!("worker", "voice worker refused the connection ({} {})", u16::from(f.code), f.reason);
|
||||
}
|
||||
return None;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
None
|
||||
})
|
||||
.await;
|
||||
let Ok(Some(hello_ok)) = hello_ok else { return false };
|
||||
|
||||
let (tx, mut rx) = mpsc::unbounded_channel::<Message>();
|
||||
{
|
||||
let mut st = inner.state.lock().unwrap();
|
||||
if st.closed {
|
||||
return true;
|
||||
}
|
||||
st.conn = Some(tx);
|
||||
}
|
||||
debug_log!("worker", "voice worker ready stt={} tts={}", hello_ok.stt_ready, hello_ok.tts_ready);
|
||||
inner.ready.send_replace(Some(hello_ok.clone()));
|
||||
let _ = inner.events.send(WorkerEvent::Ready(hello_ok));
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
out = rx.recv() => match out {
|
||||
Some(Message::Close(_)) | None => { let _ = write.close().await; break; }
|
||||
Some(m) => if write.send(m).await.is_err() { break },
|
||||
},
|
||||
msg = read.next() => match msg {
|
||||
Some(Ok(Message::Binary(b))) => on_binary(inner, &b),
|
||||
Some(Ok(Message::Text(t))) => on_text(inner, t.as_str()),
|
||||
Some(Ok(Message::Close(_))) | Some(Err(_)) | None => break,
|
||||
Some(Ok(_)) => {}
|
||||
},
|
||||
}
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
fn on_binary(inner: &Inner, b: &[u8]) {
|
||||
if b.first() != Some(&KIND_TTS) {
|
||||
return;
|
||||
}
|
||||
if let Some(f) = decode_tts_frame(b) {
|
||||
if inner.state.lock().unwrap().open_tts.contains(&f.id) {
|
||||
let _ = inner.events.send(WorkerEvent::TtsAudio { id: f.id, seq: f.seq, pcm: f.pcm });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn on_text(inner: &Inner, t: &str) {
|
||||
let Ok(v) = serde_json::from_str::<Value>(t) else { return };
|
||||
let kind = v["type"].as_str().unwrap_or_default().to_owned();
|
||||
let ev = match kind.as_str() {
|
||||
"speech_start" => serde_json::from_value(v).ok().map(WorkerEvent::SpeechStart),
|
||||
"partial" => serde_json::from_value(v).ok().map(WorkerEvent::Partial),
|
||||
"final" => serde_json::from_value(v).ok().map(WorkerEvent::Final),
|
||||
"tts_done" => serde_json::from_value::<TtsDone>(v).ok().and_then(|d| {
|
||||
inner.state.lock().unwrap().open_tts.remove(&d.id).then_some(WorkerEvent::TtsDone(d))
|
||||
}),
|
||||
"error" => serde_json::from_value::<WorkerError>(v).ok().map(|e| {
|
||||
debug_log!("worker", "voice worker error {}", e.code);
|
||||
// A request the worker refused outright never gets a tts_done.
|
||||
if e.code == "tts_unavailable" {
|
||||
if let Some(id) = &e.id {
|
||||
if inner.state.lock().unwrap().open_tts.remove(id) {
|
||||
let _ = inner.events.send(WorkerEvent::Error(e.clone()));
|
||||
return WorkerEvent::TtsDone(TtsDone { id: id.clone(), chunks: 0, cancelled: false, failed: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
WorkerEvent::Error(e)
|
||||
}),
|
||||
_ => None,
|
||||
};
|
||||
if let Some(ev) = ev {
|
||||
let _ = inner.events.send(ev);
|
||||
}
|
||||
}
|
||||
224
app/src/worker_process.rs
Normal file
224
app/src/worker_process.rs
Normal file
@@ -0,0 +1,224 @@
|
||||
//! Starts the Python voice worker as a child process and restarts it if it dies.
|
||||
//!
|
||||
//! Each app launch makes a new random token and hands it over in the environment
|
||||
//! (`JARVIS_VOICE_TOKEN`), never on the command line. The worker listens on 127.0.0.1 only.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::PathBuf;
|
||||
use std::process::Stdio;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use base64::Engine;
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
use tokio::process::{Child, Command};
|
||||
use tokio::sync::{broadcast, Notify};
|
||||
|
||||
use crate::{debug_log, error_log};
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct WorkerProcessOptions {
|
||||
pub python: String,
|
||||
pub cwd: PathBuf,
|
||||
pub args: Vec<String>,
|
||||
/// Fixed port; 0 picks a free one once at start (kept across restarts).
|
||||
pub port: u16,
|
||||
/// Extra environment for the worker (over the app's own).
|
||||
pub env: HashMap<String, String>,
|
||||
pub restart_delay: Duration,
|
||||
pub max_restart_delay: Duration,
|
||||
}
|
||||
|
||||
impl WorkerProcessOptions {
|
||||
pub fn new(python: impl Into<String>, cwd: impl Into<PathBuf>, args: Vec<String>) -> Self {
|
||||
Self {
|
||||
python: python.into(),
|
||||
cwd: cwd.into(),
|
||||
args,
|
||||
port: 0,
|
||||
env: HashMap::new(),
|
||||
restart_delay: Duration::from_secs(1),
|
||||
max_restart_delay: Duration::from_secs(30),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum ProcessEvent {
|
||||
/// The worker printed READY: models are loaded and it accepts connections.
|
||||
Ready,
|
||||
Exited { code: Option<i32> },
|
||||
}
|
||||
|
||||
pub fn free_port() -> std::io::Result<u16> {
|
||||
Ok(std::net::TcpListener::bind("127.0.0.1:0")?.local_addr()?.port())
|
||||
}
|
||||
|
||||
struct Shared {
|
||||
stopping: bool,
|
||||
child_pid: Option<u32>,
|
||||
restarts: u32,
|
||||
}
|
||||
|
||||
pub struct WorkerProcess {
|
||||
opts: WorkerProcessOptions,
|
||||
pub token: String,
|
||||
port: u16,
|
||||
shared: Arc<Mutex<Shared>>,
|
||||
stop: Arc<Notify>,
|
||||
events: broadcast::Sender<ProcessEvent>,
|
||||
task: Mutex<Option<tokio::task::JoinHandle<()>>>,
|
||||
}
|
||||
|
||||
impl WorkerProcess {
|
||||
pub fn new(opts: WorkerProcessOptions) -> std::io::Result<Self> {
|
||||
let port = if opts.port == 0 { free_port()? } else { opts.port };
|
||||
let mut raw = [0u8; 32];
|
||||
rand::fill(&mut raw);
|
||||
let token = base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(raw);
|
||||
Ok(Self {
|
||||
opts,
|
||||
token,
|
||||
port,
|
||||
shared: Arc::new(Mutex::new(Shared { stopping: false, child_pid: None, restarts: 0 })),
|
||||
stop: Arc::new(Notify::new()),
|
||||
events: broadcast::channel(64).0,
|
||||
task: Mutex::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn url(&self) -> String {
|
||||
format!("ws://127.0.0.1:{}", self.port)
|
||||
}
|
||||
|
||||
pub fn port(&self) -> u16 {
|
||||
self.port
|
||||
}
|
||||
|
||||
pub fn restarts(&self) -> u32 {
|
||||
self.shared.lock().unwrap().restarts
|
||||
}
|
||||
|
||||
pub fn subscribe(&self) -> broadcast::Receiver<ProcessEvent> {
|
||||
self.events.subscribe()
|
||||
}
|
||||
|
||||
/// Spawn the worker and keep it running until `stop()`.
|
||||
pub fn start(&self) {
|
||||
let (opts, token, port) = (self.opts.clone(), self.token.clone(), self.port);
|
||||
let (shared, stop, events) = (self.shared.clone(), self.stop.clone(), self.events.clone());
|
||||
let task = tokio::spawn(async move {
|
||||
let mut delay = opts.restart_delay;
|
||||
loop {
|
||||
let started = Instant::now();
|
||||
let mut child = match spawn(&opts, &token, port) {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
error_log!("worker", "voice worker could not start: {e}");
|
||||
let _ = events.send(ProcessEvent::Exited { code: None });
|
||||
if wait_or_stop(&stop, &shared, delay).await {
|
||||
return;
|
||||
}
|
||||
delay = (delay * 2).min(opts.max_restart_delay);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
shared.lock().unwrap().child_pid = child.id();
|
||||
debug_log!("worker", "voice worker started pid={:?} port={port}", child.id());
|
||||
let mut ready_rx = events.subscribe();
|
||||
// The worker's own logs hold timings and counts only (voice_worker.spec.md).
|
||||
if let Some(out) = child.stdout.take() {
|
||||
let events = events.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut lines = BufReader::new(out).lines();
|
||||
while let Ok(Some(line)) = lines.next_line().await {
|
||||
if line.starts_with("READY ") {
|
||||
debug_log!("worker", "voice worker READY after {} ms", started.elapsed().as_millis());
|
||||
let _ = events.send(ProcessEvent::Ready);
|
||||
} else {
|
||||
debug_log!("worker", "{line}");
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
if let Some(err) = child.stderr.take() {
|
||||
tokio::spawn(async move {
|
||||
let mut lines = BufReader::new(err).lines();
|
||||
while let Ok(Some(line)) = lines.next_line().await {
|
||||
debug_log!("worker", "{line}");
|
||||
}
|
||||
});
|
||||
}
|
||||
let status = tokio::select! {
|
||||
s = child.wait() => s.ok(),
|
||||
_ = stop.notified() => {
|
||||
let _ = child.kill().await;
|
||||
let code = child.wait().await.ok().and_then(|s| s.code());
|
||||
let _ = events.send(ProcessEvent::Exited { code });
|
||||
return;
|
||||
}
|
||||
};
|
||||
// A worker that got as far as READY resets the backoff.
|
||||
while let Ok(ev) = ready_rx.try_recv() {
|
||||
if ev == ProcessEvent::Ready {
|
||||
delay = opts.restart_delay;
|
||||
}
|
||||
}
|
||||
let code = status.and_then(|s| s.code());
|
||||
shared.lock().unwrap().child_pid = None;
|
||||
let _ = events.send(ProcessEvent::Exited { code });
|
||||
if shared.lock().unwrap().stopping {
|
||||
return;
|
||||
}
|
||||
error_log!("worker", "voice worker exited (code {code:?}); restarting in {} ms", delay.as_millis());
|
||||
if wait_or_stop(&stop, &shared, delay).await {
|
||||
return;
|
||||
}
|
||||
shared.lock().unwrap().restarts += 1;
|
||||
delay = (delay * 2).min(opts.max_restart_delay);
|
||||
}
|
||||
});
|
||||
*self.task.lock().unwrap() = Some(task);
|
||||
}
|
||||
|
||||
/// Stop the worker and do not restart it. Returns once it has exited.
|
||||
pub async fn stop(&self) {
|
||||
self.shared.lock().unwrap().stopping = true;
|
||||
self.stop.notify_one();
|
||||
let task = self.task.lock().unwrap().take();
|
||||
if let Some(t) = task {
|
||||
let _ = t.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Sleep `delay` unless stopped meanwhile (true = stopped).
|
||||
async fn wait_or_stop(stop: &Notify, shared: &Mutex<Shared>, delay: Duration) -> bool {
|
||||
if shared.lock().unwrap().stopping {
|
||||
return true;
|
||||
}
|
||||
tokio::select! {
|
||||
_ = tokio::time::sleep(delay) => shared.lock().unwrap().stopping,
|
||||
_ = stop.notified() => true,
|
||||
}
|
||||
}
|
||||
|
||||
fn spawn(opts: &WorkerProcessOptions, token: &str, port: u16) -> std::io::Result<Child> {
|
||||
let mut cmd = Command::new(&opts.python);
|
||||
cmd.args(["-m", "voice_worker", "--host", "127.0.0.1", "--port", &port.to_string()])
|
||||
.args(&opts.args)
|
||||
.current_dir(&opts.cwd)
|
||||
.envs(&opts.env)
|
||||
.env("JARVIS_VOICE_TOKEN", token)
|
||||
.env("PYTHONUNBUFFERED", "1")
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.kill_on_drop(true);
|
||||
#[cfg(windows)]
|
||||
{
|
||||
const CREATE_NO_WINDOW: u32 = 0x0800_0000;
|
||||
cmd.creation_flags(CREATE_NO_WINDOW);
|
||||
}
|
||||
cmd.spawn()
|
||||
}
|
||||
@@ -1,67 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { ChatAccessError, ChatChannels, hasMessageContentIntent, MESSAGE_LIMIT, splitMessage } from "../src/core/chat.js";
|
||||
|
||||
describe("splitMessage", () => {
|
||||
it("leaves short text alone", () => {
|
||||
expect(splitMessage("안녕하세요")).toEqual(["안녕하세요"]);
|
||||
});
|
||||
|
||||
it("breaks at the last newline, then space, within the limit and loses only the break", () => {
|
||||
const text = "aaaa bbbb\ncccc dddd";
|
||||
const parts = splitMessage(text, 12);
|
||||
expect(parts).toEqual(["aaaa bbbb", "cccc dddd"]);
|
||||
expect(splitMessage("aaaa bbbb cccc", 10)).toEqual(["aaaa bbbb", "cccc"]);
|
||||
});
|
||||
|
||||
it("cuts hard when there is no break, keeping emoji whole", () => {
|
||||
const text = "x".repeat(MESSAGE_LIMIT - 1) + "😀" + "y";
|
||||
const parts = splitMessage(text);
|
||||
expect(parts.every((p) => p.length <= MESSAGE_LIMIT)).toBe(true);
|
||||
expect(parts.join("")).toBe(text);
|
||||
expect(parts[1]!.startsWith("😀")).toBe(true);
|
||||
});
|
||||
|
||||
it("every part fits for long mixed text", () => {
|
||||
const text = Array.from({ length: 900 }, (_, i) => `줄 ${i} 내용입니다`).join("\n");
|
||||
const parts = splitMessage(text);
|
||||
expect(parts.length).toBeGreaterThan(1);
|
||||
expect(parts.every((p) => p.length > 0 && p.length <= MESSAGE_LIMIT)).toBe(true);
|
||||
expect(parts.join("\n")).toBe(text);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ChatChannels", () => {
|
||||
const ch = new ChatChannels([
|
||||
{ id: "111111111111111111", name: "일정", description: "", read: true, send: false },
|
||||
{ id: "222222222222222222", name: "공지", description: "", read: false, send: true },
|
||||
]);
|
||||
|
||||
it("follows each channel's read and send switches", () => {
|
||||
expect(ch.canRead("111111111111111111")).toBe(true);
|
||||
expect(ch.canSend("111111111111111111")).toBe(false);
|
||||
expect(ch.canRead("222222222222222222")).toBe(false);
|
||||
expect(ch.canSend("222222222222222222")).toBe(true);
|
||||
expect(() => ch.assertSend("111111111111111111")).toThrow(ChatAccessError);
|
||||
expect(() => ch.assertRead("222222222222222222")).toThrow(ChatAccessError);
|
||||
});
|
||||
|
||||
it("refuses channels that are not registered", () => {
|
||||
expect(ch.canRead("333333333333333333")).toBe(false);
|
||||
expect(() => ch.assertRead("333333333333333333")).toThrow(/not a registered/);
|
||||
expect(() => ch.assertSend("333333333333333333")).toThrow(/not a registered/);
|
||||
});
|
||||
|
||||
it("needs the message intents only when some channel is readable", () => {
|
||||
expect(ch.anyReadable).toBe(true);
|
||||
expect(new ChatChannels([{ id: "1".repeat(18), name: "", description: "", read: false, send: true }]).anyReadable)
|
||||
.toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("hasMessageContentIntent", () => {
|
||||
it("accepts the full and the limited intent flag", () => {
|
||||
expect(hasMessageContentIntent(1 << 18)).toBe(true);
|
||||
expect(hasMessageContentIntent(1 << 19)).toBe(true);
|
||||
expect(hasMessageContentIntent((1 << 15) | (1 << 13))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1,26 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { decodeTtsFrame, encodeAudioFrame, KIND_AUDIO, KIND_TTS } from "../src/core/frames.js";
|
||||
|
||||
describe("binary frames (voice_worker/protocol.py layout)", () => {
|
||||
it("audio frame is kind, id length, UTF-8 id, then PCM", () => {
|
||||
const pcm = Buffer.from([1, 0, 2, 0]);
|
||||
const f = encodeAudioFrame("유저", pcm);
|
||||
const id = Buffer.from("유저", "utf8");
|
||||
expect([...f]).toEqual([KIND_AUDIO, id.length, ...id, 1, 0, 2, 0]);
|
||||
});
|
||||
|
||||
it("rejects ids that do not fit in a byte", () => {
|
||||
expect(() => encodeAudioFrame("", Buffer.alloc(2))).toThrow();
|
||||
expect(() => encodeAudioFrame("x".repeat(256), Buffer.alloc(2))).toThrow();
|
||||
});
|
||||
|
||||
it("decodes a TTS frame and refuses truncated ones", () => {
|
||||
const id = Buffer.from("r1");
|
||||
const seq = Buffer.alloc(4);
|
||||
seq.writeUInt32LE(7);
|
||||
const frame = Buffer.concat([Buffer.from([KIND_TTS, id.length]), id, seq, Buffer.from([9, 9])]);
|
||||
expect(decodeTtsFrame(frame)).toEqual({ id: "r1", seq: 7, pcm: Buffer.from([9, 9]) });
|
||||
expect(decodeTtsFrame(frame.subarray(0, 5))).toBeNull();
|
||||
expect(decodeTtsFrame(Buffer.from([KIND_AUDIO, 1, 65, 0, 0, 0, 0]))).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -1,317 +0,0 @@
|
||||
/**
|
||||
* The Discord voice core end to end on real Discord, with the real voice worker on the GPU.
|
||||
*
|
||||
* Two bots in one voice channel: the "Jarvis" bot runs the core; the "speaker" bot plays recorded
|
||||
* Korean speech into the channel, records what Jarvis says, and writes in the channel's chat.
|
||||
* Opt-in; it refuses to run if a person is in the channel.
|
||||
*
|
||||
* JARVIS_LIVE_DISCORD=1 JARVIS_LIVE_BOT_TOKEN=... JARVIS_LIVE_SPEAKER_TOKEN=... \
|
||||
* JARVIS_LIVE_GUILD=<server id> JARVIS_LIVE_VOICE_CHANNEL=<voice channel id> \
|
||||
* JARVIS_VOICE_PYTHON=<worker venv python> JARVIS_MELO_PYTHON=<MeloTTS venv python> \
|
||||
* npx vitest run test/live
|
||||
*
|
||||
* On Linux, put the pip CUDA libraries on LD_LIBRARY_PATH (nvidia/cublas/lib, nvidia/cudnn/lib).
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { Client, Events, GatewayIntentBits, REST, Routes, type Message, type VoiceBasedChannel } from "discord.js";
|
||||
import {
|
||||
AudioPlayerStatus,
|
||||
createAudioPlayer,
|
||||
createAudioResource,
|
||||
EndBehaviorType,
|
||||
entersState,
|
||||
joinVoiceChannel,
|
||||
StreamType,
|
||||
VoiceConnectionStatus,
|
||||
type VoiceConnection,
|
||||
} from "@discordjs/voice";
|
||||
import OpusScript from "opusscript";
|
||||
import { Readable } from "node:stream";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { DISCORD_PCM, DiscordCore, type HeardFinal } from "../../src/core/discordCore.js";
|
||||
import { parseSettings } from "../../src/core/settings.js";
|
||||
import { VoiceWorkerClient, type Final } from "../../src/core/workerClient.js";
|
||||
import { VoiceWorkerProcess } from "../../src/core/workerProcess.js";
|
||||
import type { PlaybackEnd } from "../../src/core/playback.js";
|
||||
import type { ChatMessage } from "../../src/core/chat.js";
|
||||
|
||||
const env = process.env;
|
||||
const LIVE = env.JARVIS_LIVE_DISCORD === "1";
|
||||
const GUILD = env.JARVIS_LIVE_GUILD!;
|
||||
const VOICE = env.JARVIS_LIVE_VOICE_CHANNEL!;
|
||||
const COMMAND = "jarvis-live-test";
|
||||
const SAMPLES = fileURLToPath(new URL("../../../scripts/m0/assets/ko_samples/", import.meta.url));
|
||||
const SRC = fileURLToPath(new URL("../../../src", import.meta.url));
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
async function until(pred: () => boolean, ms: number, what: string) {
|
||||
const end = Date.now() + ms;
|
||||
while (!pred()) {
|
||||
if (Date.now() > end) throw new Error(`timed out waiting for ${what}`);
|
||||
await sleep(25);
|
||||
}
|
||||
}
|
||||
|
||||
function cer(ref: string, hyp: string): number {
|
||||
const norm = (s: string) => [...s.replace(/[^\p{L}\p{N}]+/gu, "")];
|
||||
const a = norm(ref);
|
||||
const b = norm(hyp);
|
||||
let prev = Array.from({ length: b.length + 1 }, (_, j) => j);
|
||||
for (let i = 1; i <= a.length; i++) {
|
||||
const cur = [i];
|
||||
for (let j = 1; j <= b.length; j++) {
|
||||
cur.push(Math.min(prev[j]! + 1, cur[j - 1]! + 1, prev[j - 1]! + (a[i - 1] === b[j - 1] ? 0 : 1)));
|
||||
}
|
||||
prev = cur;
|
||||
}
|
||||
return prev[b.length]! / Math.max(1, a.length);
|
||||
}
|
||||
|
||||
/** Seconds into 48 kHz stereo PCM where the last sound above -34 dBFS ends (the end of speech). */
|
||||
function speechEndSeconds(pcm: Buffer): number {
|
||||
for (let k = pcm.length / 4 - 1; k >= 0; k--) if (Math.abs(pcm.readInt16LE(k * 4)) > 655) return (k + 1) / 48000;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** A 16 kHz mono PCM16 WAV as 48 kHz stereo PCM16 (Discord's format). */
|
||||
function wavTo48kStereo(path: string): Buffer {
|
||||
const wav = readFileSync(path);
|
||||
let off = 12;
|
||||
let data: Buffer | null = null;
|
||||
while (off + 8 <= wav.length) {
|
||||
const id = wav.toString("ascii", off, off + 4);
|
||||
const size = wav.readUInt32LE(off + 4);
|
||||
if (id === "data") data = wav.subarray(off + 8, off + 8 + size);
|
||||
off += 8 + size + (size % 2);
|
||||
}
|
||||
if (!data) throw new Error(`no data chunk in ${path}`);
|
||||
const n = data.length / 2;
|
||||
const out = Buffer.alloc(n * 3 * 4);
|
||||
for (let k = 0; k < n * 3; k++) {
|
||||
const pos = k / 3;
|
||||
const i = Math.floor(pos);
|
||||
const a = data.readInt16LE(i * 2);
|
||||
const b = i + 1 < n ? data.readInt16LE((i + 1) * 2) : a;
|
||||
const v = Math.round(a + (b - a) * (pos - i));
|
||||
out.writeInt16LE(v, k * 4);
|
||||
out.writeInt16LE(v, k * 4 + 2);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe.skipIf(!LIVE)("Discord voice core on real Discord", () => {
|
||||
let workerProcess: VoiceWorkerProcess;
|
||||
let worker: VoiceWorkerClient;
|
||||
let core: DiscordCore;
|
||||
let speaker: Client;
|
||||
let speakerVoice: VoiceConnection;
|
||||
let jarvisId = "";
|
||||
const finals: HeardFinal[] = [];
|
||||
const speakEnds: PlaybackEnd[] = [];
|
||||
const chats: ChatMessage[] = [];
|
||||
/** Jarvis's voice as the speaker bot hears it: decoded PCM and packet times. */
|
||||
const heard = { pcm: [] as Buffer[], lastPacketAt: 0, packets: 0 };
|
||||
|
||||
beforeAll(async () => {
|
||||
// The speaker bot first: it checks the channel is free of people before anything joins.
|
||||
speaker = new Client({ intents: [GatewayIntentBits.Guilds, GatewayIntentBits.GuildVoiceStates,
|
||||
GatewayIntentBits.GuildMessages, GatewayIntentBits.MessageContent] });
|
||||
await speaker.login(env.JARVIS_LIVE_SPEAKER_TOKEN);
|
||||
await new Promise<void>((r) => (speaker.isReady() ? r() : speaker.once(Events.ClientReady, () => r())));
|
||||
const guild = await speaker.guilds.fetch(GUILD);
|
||||
const people = guild.voiceStates.cache.filter((v) => v.channelId === VOICE && !v.member?.user.bot);
|
||||
if (people.size) throw new Error(`refusing to test: ${people.size} people are in the voice channel`);
|
||||
// Either bot already in a voice channel of this server is in use elsewhere; joining would move it.
|
||||
const jarvisApp = (await new REST({ version: "10" }).setToken(env.JARVIS_LIVE_BOT_TOKEN!)
|
||||
.get(Routes.currentApplication())) as { id: string };
|
||||
for (const id of [speaker.user!.id, jarvisApp.id]) {
|
||||
if (guild.voiceStates.cache.get(id)?.channelId) throw new Error(`refusing to test: bot ${id} is in a voice channel`);
|
||||
}
|
||||
|
||||
workerProcess = new VoiceWorkerProcess({ python: env.JARVIS_VOICE_PYTHON!, cwd: SRC,
|
||||
args: env.JARVIS_MELO_PYTHON ? ["--melo-python", env.JARVIS_MELO_PYTHON] : [] });
|
||||
const workerReady = new Promise<void>((r) => workerProcess.once("ready", () => r()));
|
||||
await workerProcess.start();
|
||||
await workerReady;
|
||||
worker = new VoiceWorkerClient({ url: workerProcess.url, token: workerProcess.token, audio: DISCORD_PCM,
|
||||
tts: DISCORD_PCM, stt: { hotwords: ["자비스"] } });
|
||||
worker.start();
|
||||
await worker.waitReady(10_000);
|
||||
|
||||
const settings = parseSettings({
|
||||
discord: { guildId: GUILD, voiceChannelId: VOICE, commandName: COMMAND },
|
||||
// The "person" here is a bot, so bots must be heard.
|
||||
voice: { listenToBots: true },
|
||||
// The voice channel's own chat stands in for a registered chat channel.
|
||||
chatChannels: [{ id: VOICE, name: "테스트", description: "라이브 테스트", read: true, send: true }],
|
||||
});
|
||||
core = new DiscordCore(settings, env.JARVIS_LIVE_BOT_TOKEN!, worker);
|
||||
core.on("final", (f) => finals.push(f));
|
||||
core.on("speak_end", (e) => speakEnds.push(e));
|
||||
core.on("chat", (c) => chats.push(c));
|
||||
core.on("ready", (r) => (jarvisId = r.userId));
|
||||
await core.start(); // logs in, registers the command, auto-joins VOICE, posts the notice
|
||||
expect(core.voiceChannelId).toBe(VOICE);
|
||||
|
||||
const channel = (await speaker.channels.fetch(VOICE)) as VoiceBasedChannel;
|
||||
speakerVoice = joinVoiceChannel({ channelId: VOICE, guildId: GUILD, adapterCreator: channel.guild.voiceAdapterCreator,
|
||||
selfDeaf: false, group: "speaker" });
|
||||
await entersState(speakerVoice, VoiceConnectionStatus.Ready, 20_000);
|
||||
speakerVoice.receiver.speaking.on("start", (userId) => {
|
||||
if (userId !== jarvisId) return;
|
||||
const stream = speakerVoice.receiver.subscribe(userId, { end: { behavior: EndBehaviorType.Manual } });
|
||||
if (stream.listenerCount("data")) return;
|
||||
const dec = new OpusScript(48000, 2, OpusScript.Application.VOIP);
|
||||
stream.on("data", (p: Buffer) => {
|
||||
heard.packets++;
|
||||
heard.lastPacketAt = Date.now();
|
||||
try {
|
||||
heard.pcm.push(Buffer.from(dec.decode(p)));
|
||||
} catch { /* corrupt packet */ }
|
||||
});
|
||||
});
|
||||
}, 240_000);
|
||||
|
||||
afterAll(async () => {
|
||||
try {
|
||||
const rest = new REST({ version: "10" }).setToken(env.JARVIS_LIVE_BOT_TOKEN!);
|
||||
const app = (await rest.get(Routes.currentApplication())) as { id: string };
|
||||
const cmds = (await rest.get(Routes.applicationGuildCommands(app.id, GUILD))) as { id: string; name: string }[];
|
||||
for (const c of cmds.filter((c) => c.name === COMMAND)) {
|
||||
await rest.delete(Routes.applicationGuildCommand(app.id, GUILD, c.id));
|
||||
}
|
||||
} catch { /* best effort */ }
|
||||
speakerVoice?.destroy();
|
||||
await core?.close();
|
||||
await speaker?.destroy();
|
||||
worker?.close();
|
||||
await workerProcess?.stop();
|
||||
}, 60_000);
|
||||
|
||||
it("registered the slash command in the server", async () => {
|
||||
const rest = new REST({ version: "10" }).setToken(env.JARVIS_LIVE_BOT_TOKEN!);
|
||||
const app = (await rest.get(Routes.currentApplication())) as { id: string };
|
||||
const cmds = (await rest.get(Routes.applicationGuildCommands(app.id, GUILD))) as
|
||||
{ name: string; options: { name: string }[] }[];
|
||||
const cmd = cmds.find((c) => c.name === COMMAND);
|
||||
expect(cmd?.options.map((o) => o.name).sort()).toEqual(["join", "leave", "stop"]);
|
||||
});
|
||||
|
||||
it("hears each recorded sentence played into the channel as a final transcript", async () => {
|
||||
const refs = JSON.parse(readFileSync(`${SAMPLES}refs.json`, "utf8")).samples as { file: string; text: string }[];
|
||||
const player = createAudioPlayer();
|
||||
speakerVoice.subscribe(player);
|
||||
const speakerId = speaker.user!.id;
|
||||
const results: { ref: string; hyp: string; cer: number; speechEndToFinalMs: number; endReason: string }[] = [];
|
||||
for (const ref of refs.slice(0, 5)) {
|
||||
const before = finals.length;
|
||||
const pcm = wavTo48kStereo(SAMPLES + ref.file);
|
||||
player.play(createAudioResource(Readable.from([pcm]), { inputType: StreamType.Raw }));
|
||||
await entersState(player, AudioPlayerStatus.Playing, 5000);
|
||||
const speechEndsAt = Date.now() + speechEndSeconds(pcm) * 1000;
|
||||
let finalAt = 0;
|
||||
await until(() => {
|
||||
const ok = finals.slice(before).some((f) => f.speakerId === speakerId);
|
||||
if (ok && !finalAt) finalAt = Date.now();
|
||||
return ok;
|
||||
}, 20_000, `final for ${ref.file}`);
|
||||
await entersState(player, AudioPlayerStatus.Idle, 20_000);
|
||||
await sleep(800);
|
||||
const mine = finals.slice(before).filter((f) => f.speakerId === speakerId);
|
||||
const hyp = mine.map((f) => f.text).join(" ");
|
||||
results.push({ ref: ref.text, hyp, cer: cer(ref.text, hyp), speechEndToFinalMs: finalAt - speechEndsAt,
|
||||
endReason: mine.map((f) => f.endReason).join(",") });
|
||||
expect(mine[0]!.speakerName.length).toBeGreaterThan(0);
|
||||
}
|
||||
console.log("live STT results", JSON.stringify(results, null, 1));
|
||||
const mean = results.reduce((s, r) => s + r.cer, 0) / results.length;
|
||||
expect(results.every((r) => r.hyp.length > 0)).toBe(true);
|
||||
expect(mean).toBeLessThan(0.15);
|
||||
}, 180_000);
|
||||
|
||||
it("speaks a reply the other side hears, and the STT reads it back", async () => {
|
||||
const text = "안녕하세요, 저는 자비스입니다. 디스코드 음성 연결을 확인하고 있어요.";
|
||||
heard.pcm.length = 0;
|
||||
const before = speakEnds.length;
|
||||
const id = core.say(text);
|
||||
expect(id).not.toBeNull();
|
||||
await until(() => speakEnds.length > before, 30_000, "speech to end");
|
||||
expect(speakEnds.at(-1)).toMatchObject({ id, cancelled: false, failed: false });
|
||||
await sleep(500);
|
||||
const pcm = Buffer.concat(heard.pcm);
|
||||
const seconds = pcm.length / (48000 * 4);
|
||||
expect(seconds).toBeGreaterThan(2);
|
||||
// Read Jarvis's voice back through the STT on a second worker session.
|
||||
const echo = new VoiceWorkerClient({ url: workerProcess.url, token: workerProcess.token, audio: DISCORD_PCM,
|
||||
tts: DISCORD_PCM, stt: { partials: false } });
|
||||
echo.start();
|
||||
await echo.waitReady(5000);
|
||||
const got: Final[] = [];
|
||||
echo.on("final", (f) => got.push(f));
|
||||
const frame = 3840;
|
||||
for (let k = 0; k < pcm.length; k += frame) echo.sendAudio("echo", pcm.subarray(k, k + frame));
|
||||
for (let k = 0; k < 25; k++) echo.sendAudio("echo", Buffer.alloc(frame)); // 0.5 s of silence
|
||||
echo.speakingEnd("echo");
|
||||
await sleep(3000);
|
||||
echo.close();
|
||||
const hyp = got.map((f) => f.text).join(" ");
|
||||
console.log("live TTS read back", JSON.stringify({ seconds, hyp, cer: cer(text, hyp) }));
|
||||
expect(cer(text, hyp)).toBeLessThan(0.2);
|
||||
}, 90_000);
|
||||
|
||||
it("stop silences it within a few frames and cancels the rest", async () => {
|
||||
const text = "이 문장은 정지 명령을 시험하려고 길게 말하는 첫 번째 문장입니다. "
|
||||
+ "두 번째 문장도 꽤 길게 이어집니다. 세 번째 문장까지 들리면 정지가 안 된 것입니다. "
|
||||
+ "네 번째 문장은 절대 들리면 안 됩니다.";
|
||||
const before = speakEnds.length;
|
||||
let started = false;
|
||||
core.once("speak_start", () => (started = true));
|
||||
const id = core.say(text);
|
||||
await until(() => started, 30_000, "speech to start");
|
||||
await sleep(1500);
|
||||
const stopAt = Date.now();
|
||||
expect(core.stop()).toBe(1);
|
||||
await sleep(1500);
|
||||
expect(speakEnds.slice(before)).toEqual([expect.objectContaining({ id, cancelled: true })]);
|
||||
const tail = heard.lastPacketAt - stopAt;
|
||||
console.log("live stop", JSON.stringify({ lastPacketAfterStopMs: tail }));
|
||||
// The player sends no padding after a forced stop; allow for network delivery.
|
||||
expect(tail).toBeLessThan(400);
|
||||
expect(core.speaking).toBe(false);
|
||||
}, 60_000);
|
||||
|
||||
it("reads the registered chat channel live and in history, and sends to it without pings", async () => {
|
||||
const channel = (await speaker.channels.fetch(VOICE)) as VoiceBasedChannel;
|
||||
const marker = `라이브 테스트 메시지 ${Date.now()}`;
|
||||
await channel.send(marker);
|
||||
await until(() => chats.some((c) => c.content === marker), 10_000, "chat event");
|
||||
expect(chats.find((c) => c.content === marker)).toMatchObject({ channelId: VOICE, authorId: speaker.user!.id,
|
||||
authorIsBot: true, fromSelf: false });
|
||||
|
||||
const history = await core.readChat(VOICE, 20);
|
||||
expect(history.some((m) => m.content === marker && !m.fromSelf)).toBe(true);
|
||||
// The transcription notice from joining is there, written by Jarvis.
|
||||
expect(history.some((m) => m.fromSelf && m.content.includes("받아 적습니다"))).toBe(true);
|
||||
|
||||
const seen: Message[] = [];
|
||||
speaker.on(Events.MessageCreate, (m) => void (m.author.id === jarvisId && seen.push(m)));
|
||||
const reply = `자비스 답장 @everyone ${Date.now()}`;
|
||||
const ids = await core.sendChat(VOICE, reply);
|
||||
await until(() => seen.some((m) => ids.includes(m.id)), 10_000, "sent message");
|
||||
const sent = seen.find((m) => ids.includes(m.id))!;
|
||||
expect(sent.content).toBe(reply);
|
||||
expect(sent.mentions.everyone).toBe(false);
|
||||
// Our own message does not come back as a chat event.
|
||||
await sleep(500);
|
||||
expect(chats.some((c) => c.content === reply)).toBe(false);
|
||||
}, 60_000);
|
||||
|
||||
it("leaves the voice channel", async () => {
|
||||
const guild = await speaker.guilds.fetch(GUILD);
|
||||
expect(core.leave()).toBe(true);
|
||||
await until(() => guild.voiceStates.cache.get(jarvisId)?.channelId == null, 10_000, "Jarvis to leave");
|
||||
expect(core.voiceChannelId).toBeNull();
|
||||
expect(core.say("아무도 못 듣는 말")).toBeNull();
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -1,121 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { Playback, type PlaybackEnd } from "../src/core/playback.js";
|
||||
|
||||
/** A fake worker and speaker: records what was asked of them and what was played. */
|
||||
function setup() {
|
||||
const sent: { id: string; text: string; more: boolean }[] = [];
|
||||
const cancelled: string[] = [];
|
||||
const played: string[] = [];
|
||||
let stops = 0;
|
||||
const pb = new Playback(
|
||||
{ tts: (id, text, more = false) => (sent.push({ id, text, more }), true), ttsCancel: (id) => void cancelled.push(id) },
|
||||
{ play: (pcm) => void played.push(pcm.toString()), stop: () => void stops++ },
|
||||
);
|
||||
const ends: PlaybackEnd[] = [];
|
||||
const starts: string[] = [];
|
||||
pb.on("end", (e) => ends.push(e));
|
||||
pb.on("start", (id) => starts.push(id));
|
||||
return { pb, sent, cancelled, played, ends, starts, stops: () => stops };
|
||||
}
|
||||
|
||||
const pcm = (s: string) => Buffer.from(s);
|
||||
|
||||
describe("Playback", () => {
|
||||
it("plays the first sentence as soon as it arrives, then the next when the clip ends", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("하나. 둘.");
|
||||
expect(t.sent).toEqual([{ id, text: "하나. 둘.", more: false }]);
|
||||
t.pb.audio(id, 0, pcm("s0"));
|
||||
expect(t.played).toEqual(["s0"]);
|
||||
expect(t.starts).toEqual([id]);
|
||||
t.pb.audio(id, 1, pcm("s1"));
|
||||
expect(t.played).toEqual(["s0"]); // still playing s0
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["s0", "s1"]);
|
||||
t.pb.done({ id, chunks: 2, cancelled: false, failed: false });
|
||||
expect(t.ends).toEqual([]); // s1 is still playing
|
||||
t.pb.clipEnded();
|
||||
expect(t.ends).toEqual([{ id, played: 2, cancelled: false, failed: false }]);
|
||||
expect(t.pb.busy).toBe(false);
|
||||
});
|
||||
|
||||
it("plays requests in the order they were made", () => {
|
||||
const t = setup();
|
||||
const a = t.pb.say("a");
|
||||
const b = t.pb.say("b");
|
||||
t.pb.audio(b, 0, pcm("b0")); // b's audio arrives first
|
||||
expect(t.played).toEqual([]);
|
||||
t.pb.audio(a, 0, pcm("a0"));
|
||||
t.pb.done({ id: a, chunks: 1, cancelled: false, failed: false });
|
||||
t.pb.done({ id: b, chunks: 1, cancelled: false, failed: false });
|
||||
t.pb.clipEnded();
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["a0", "b0"]);
|
||||
expect(t.ends.map((e) => e.id)).toEqual([a, b]);
|
||||
});
|
||||
|
||||
it("keeps sentences in order even when they arrive out of order", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("x", true);
|
||||
t.pb.audio(id, 1, pcm("s1"));
|
||||
expect(t.played).toEqual([]);
|
||||
t.pb.audio(id, 0, pcm("s0"));
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["s0", "s1"]);
|
||||
});
|
||||
|
||||
it("skips a sentence that failed and still plays the rest", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("a. b. c.");
|
||||
t.pb.audio(id, 0, pcm("s0"));
|
||||
t.pb.audio(id, 2, pcm("s2"));
|
||||
t.pb.done({ id, chunks: 2, cancelled: false, failed: true });
|
||||
t.pb.clipEnded();
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["s0", "s2"]);
|
||||
expect(t.ends).toEqual([{ id, played: 2, cancelled: false, failed: true }]);
|
||||
});
|
||||
|
||||
it("stop silences the clip, cancels unfinished requests and drops the queue", () => {
|
||||
const t = setup();
|
||||
const a = t.pb.say("a", true);
|
||||
const b = t.pb.say("b");
|
||||
t.pb.audio(a, 0, pcm("a0"));
|
||||
t.pb.done({ id: b, chunks: 0, cancelled: false, failed: false });
|
||||
expect(t.pb.stop()).toBe(2);
|
||||
expect(t.stops()).toBe(1);
|
||||
expect(t.cancelled).toEqual([a]); // b was already done at the worker
|
||||
expect(t.ends).toEqual([
|
||||
{ id: a, played: 1, cancelled: true, failed: false },
|
||||
{ id: b, played: 0, cancelled: true, failed: false },
|
||||
]);
|
||||
// Late audio for a stopped request is ignored.
|
||||
t.pb.audio(a, 1, pcm("a1"));
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["a0"]);
|
||||
expect(t.pb.busy).toBe(false);
|
||||
});
|
||||
|
||||
it("stop with nothing to say reports zero and touches nothing", () => {
|
||||
const t = setup();
|
||||
expect(t.pb.stop()).toBe(0);
|
||||
expect(t.stops()).toBe(0);
|
||||
});
|
||||
|
||||
it("text can be added to an open request", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("첫 문장.", true);
|
||||
t.pb.continueText(id, " 둘째 문장.", false);
|
||||
expect(t.sent.map((s) => [s.text, s.more])).toEqual([["첫 문장.", true], [" 둘째 문장.", false]]);
|
||||
});
|
||||
|
||||
it("a request the worker never accepted ends as failed without blocking the next", () => {
|
||||
const t = setup();
|
||||
const a = t.pb.say("a");
|
||||
const b = t.pb.say("b");
|
||||
t.pb.done({ id: a, chunks: 0, cancelled: false, failed: true });
|
||||
t.pb.audio(b, 0, pcm("b0"));
|
||||
expect(t.played).toEqual(["b0"]);
|
||||
expect(t.ends[0]).toEqual({ id: a, played: 0, cancelled: false, failed: true });
|
||||
});
|
||||
});
|
||||
@@ -1,61 +0,0 @@
|
||||
import { mkdtempSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { defaultSettings, loadSettings, parseSettings, SettingsError } from "../src/core/settings.js";
|
||||
|
||||
const ID = "123456789012345678";
|
||||
|
||||
describe("settings", () => {
|
||||
it("an empty file gives the defaults", () => {
|
||||
expect(parseSettings({})).toEqual(defaultSettings());
|
||||
});
|
||||
|
||||
it("a missing file gives the defaults", () => {
|
||||
const dir = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-settings-"));
|
||||
expect(loadSettings(join(dir, "nope.json"))).toEqual(defaultSettings());
|
||||
});
|
||||
|
||||
it("reads a full file, BOM included", () => {
|
||||
const dir = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-settings-"));
|
||||
const path = join(dir, "settings.json");
|
||||
writeFileSync(path, "" + JSON.stringify({
|
||||
discord: { guildId: ID, voiceChannelId: ID, commandName: "jarvis", ephemeralReplies: false },
|
||||
voice: { maxSpeakers: 3, listenToBots: true, joinNotice: "알림", wakeWords: [" 자비스 ", "", "javis"],
|
||||
stt: { end_silence_ms: 250 } },
|
||||
chatChannels: [{ id: ID, name: "일정", description: "일정 올리는 곳", send: false }],
|
||||
voiceWorker: { python: "/venv/bin/python", cwd: "/repo/src", args: ["--melo-python", "/m/python"] },
|
||||
somethingNew: 1,
|
||||
}));
|
||||
const s = loadSettings(path);
|
||||
expect(s.discord).toEqual({ guildId: ID, voiceChannelId: ID, commandName: "jarvis", ephemeralReplies: false });
|
||||
expect(s.voice).toEqual({ maxSpeakers: 3, listenToBots: true, joinNotice: "알림", wakeWords: ["자비스", "javis"],
|
||||
stt: { end_silence_ms: 250 } });
|
||||
expect(s.chatChannels).toEqual([{ id: ID, name: "일정", description: "일정 올리는 곳", read: true, send: false }]);
|
||||
expect(s.voiceWorker.args).toEqual(["--melo-python", "/m/python"]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[{ discord: { guildId: "abc" } }, /discord.guildId/],
|
||||
[{ discord: { commandName: "Jarvis" } }, /commandName/],
|
||||
[{ discord: { commandName: "두 단어" } }, /commandName/],
|
||||
[{ voice: { maxSpeakers: 11 } }, /maxSpeakers/],
|
||||
[{ voice: { maxSpeakers: 0 } }, /maxSpeakers/],
|
||||
[{ voice: { joinNotice: " " } }, /joinNotice/],
|
||||
[{ chatChannels: [{ id: ID }, { id: ID }] }, /twice/],
|
||||
[{ chatChannels: [{ id: "1" }] }, /chatChannels\[0\].id/],
|
||||
[{ chatChannels: {} }, /chatChannels/],
|
||||
[{ voice: [] }, /voice/],
|
||||
[[], /object/],
|
||||
])("refuses %j", (raw, message) => {
|
||||
expect(() => parseSettings(raw)).toThrow(SettingsError);
|
||||
expect(() => parseSettings(raw)).toThrow(message);
|
||||
});
|
||||
|
||||
it("reports invalid JSON with the path", () => {
|
||||
const dir = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-settings-"));
|
||||
const path = join(dir, "settings.json");
|
||||
writeFileSync(path, "{ nope");
|
||||
expect(() => loadSettings(path)).toThrow(/not valid JSON/);
|
||||
});
|
||||
});
|
||||
@@ -1,24 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { SpeakerGate } from "../src/core/speakers.js";
|
||||
|
||||
describe("SpeakerGate", () => {
|
||||
it("hears up to the limit and lets the next person in once someone stops", () => {
|
||||
const g = new SpeakerGate(2);
|
||||
expect(g.start("a")).toBe(true);
|
||||
expect(g.start("b")).toBe(true);
|
||||
expect(g.start("c")).toBe(false);
|
||||
expect(g.isActive("c")).toBe(false);
|
||||
expect(g.end("a")).toBe(true);
|
||||
expect(g.start("c")).toBe(true);
|
||||
expect(g.count).toBe(2);
|
||||
});
|
||||
|
||||
it("a speaker already heard stays heard; ending someone not heard is a no-op", () => {
|
||||
const g = new SpeakerGate(1);
|
||||
expect(g.start("a")).toBe(true);
|
||||
expect(g.start("a")).toBe(true);
|
||||
expect(g.end("z")).toBe(false);
|
||||
expect(g.end("a")).toBe(true);
|
||||
expect(g.end("a")).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"extends": "../tsconfig.json",
|
||||
"compilerOptions": { "noEmit": true, "rootDir": ".." },
|
||||
"include": ["../src", "."]
|
||||
}
|
||||
@@ -1,171 +0,0 @@
|
||||
import { once } from "node:events";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { WebSocketServer, type WebSocket } from "ws";
|
||||
import { decodeTtsFrame, encodeAudioFrame, KIND_TTS } from "../src/core/frames.js";
|
||||
import { VoiceWorkerClient, type TtsDone } from "../src/core/workerClient.js";
|
||||
|
||||
const TOKEN = "t0ken";
|
||||
const FMT = { rate: 48000, channels: 2 } as const;
|
||||
|
||||
/** A stand-in voice worker: checks the hello and records what the client sends. */
|
||||
async function fakeWorker(port = 0) {
|
||||
const wss = new WebSocketServer({ host: "127.0.0.1", port });
|
||||
await once(wss, "listening");
|
||||
const received: (Record<string, unknown> | Buffer)[] = [];
|
||||
const sockets: WebSocket[] = [];
|
||||
wss.on("connection", (ws) => {
|
||||
sockets.push(ws);
|
||||
ws.on("message", (data, isBinary) => {
|
||||
if (isBinary) return void received.push(data as Buffer);
|
||||
const msg = JSON.parse(String(data));
|
||||
received.push(msg);
|
||||
if (msg.type === "hello") {
|
||||
if (msg.token !== TOKEN) return ws.close(4401, "bad token");
|
||||
ws.send(JSON.stringify({ type: "hello_ok", stt_ready: true, tts_ready: true, stt: msg.stt }));
|
||||
}
|
||||
});
|
||||
});
|
||||
const addr = wss.address() as { port: number };
|
||||
return { wss, received, sockets, url: `ws://127.0.0.1:${addr.port}`, port: addr.port };
|
||||
}
|
||||
|
||||
const cleanups: (() => void | Promise<void>)[] = [];
|
||||
afterEach(async () => {
|
||||
for (const c of cleanups.splice(0)) await c();
|
||||
});
|
||||
|
||||
function client(url: string, extra: Partial<ConstructorParameters<typeof VoiceWorkerClient>[0]> = {}) {
|
||||
const c = new VoiceWorkerClient({ url, token: TOKEN, audio: FMT, tts: FMT, backoffMs: 20, maxBackoffMs: 50, ...extra });
|
||||
cleanups.push(() => c.close());
|
||||
return c;
|
||||
}
|
||||
|
||||
const until = async (pred: () => boolean, ms = 3000) => {
|
||||
const end = Date.now() + ms;
|
||||
while (!pred()) {
|
||||
if (Date.now() > end) throw new Error("timed out");
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
}
|
||||
};
|
||||
|
||||
describe("VoiceWorkerClient", () => {
|
||||
it("says hello with the token, formats and STT settings, then streams audio frames", async () => {
|
||||
const w = await fakeWorker();
|
||||
cleanups.push(() => void w.wss.close());
|
||||
const c = client(w.url, { stt: { hotwords: ["자비스"] } });
|
||||
c.start();
|
||||
const ok = await c.waitReady(2000);
|
||||
expect(ok.stt).toEqual({ hotwords: ["자비스"] });
|
||||
expect(w.received[0]).toEqual({ type: "hello", token: TOKEN, audio: FMT, tts: FMT, stt: { hotwords: ["자비스"] } });
|
||||
const pcm = Buffer.alloc(3840, 1);
|
||||
c.sendAudio("123", pcm);
|
||||
c.speakingEnd("123");
|
||||
c.speakerLeave("123");
|
||||
await until(() => w.received.length >= 4);
|
||||
expect(w.received[1]).toEqual(encodeAudioFrame("123", pcm));
|
||||
expect(w.received.slice(2)).toEqual([{ type: "speaking_end", speaker: "123" }, { type: "speaker_leave", speaker: "123" }]);
|
||||
});
|
||||
|
||||
it("passes worker events on and only TTS audio of requests it opened", async () => {
|
||||
const w = await fakeWorker();
|
||||
cleanups.push(() => void w.wss.close());
|
||||
const c = client(w.url);
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
const finals: unknown[] = [];
|
||||
const audio: { id: string; seq: number }[] = [];
|
||||
const done: TtsDone[] = [];
|
||||
c.on("final", (f) => finals.push(f));
|
||||
c.on("tts_audio", (a) => audio.push({ id: a.id, seq: a.seq }));
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
expect(c.tts("r1", "안녕하세요.")).toBe(true);
|
||||
await until(() => w.received.some((m) => !Buffer.isBuffer(m) && m.type === "tts"));
|
||||
const ws = w.sockets[0]!;
|
||||
const frame = (id: string, seq: number) => {
|
||||
const s = Buffer.alloc(4);
|
||||
s.writeUInt32LE(seq);
|
||||
return Buffer.concat([Buffer.from([KIND_TTS, id.length]), Buffer.from(id), s, Buffer.alloc(8)]);
|
||||
};
|
||||
ws.send(frame("r1", 0));
|
||||
ws.send(frame("zz", 0)); // not ours
|
||||
ws.send(JSON.stringify({ type: "final", speaker: "1", utt: 1, text: "네", end_reason: "silence", audio_ms: 900,
|
||||
wait_ms: 100, stt_ms: 80 }));
|
||||
ws.send(JSON.stringify({ type: "tts_done", id: "r1", chunks: 1, cancelled: false, failed: false }));
|
||||
await until(() => done.length === 1 && finals.length === 1);
|
||||
expect(audio).toEqual([{ id: "r1", seq: 0 }]);
|
||||
expect(decodeTtsFrame(frame("r1", 0))?.pcm.length).toBe(8);
|
||||
expect(done[0]).toEqual({ id: "r1", chunks: 1, cancelled: false, failed: false });
|
||||
});
|
||||
|
||||
it("a refused TTS request (no engine) ends as failed", async () => {
|
||||
const w = await fakeWorker();
|
||||
cleanups.push(() => void w.wss.close());
|
||||
const c = client(w.url);
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
const done: TtsDone[] = [];
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
c.tts("r9", "x");
|
||||
await until(() => w.received.length >= 2);
|
||||
w.sockets[0]!.send(JSON.stringify({ type: "error", code: "tts_unavailable", message: "no", id: "r9" }));
|
||||
await until(() => done.length === 1);
|
||||
expect(done[0]).toMatchObject({ id: "r9", failed: true });
|
||||
});
|
||||
|
||||
it("reconnects after the worker restarts and fails the TTS that was in flight", async () => {
|
||||
let w = await fakeWorker();
|
||||
const port = w.port;
|
||||
const c = client(w.url);
|
||||
let readies = 0;
|
||||
let disconnects = 0;
|
||||
c.on("ready", () => readies++);
|
||||
c.on("disconnected", () => disconnects++);
|
||||
const done: TtsDone[] = [];
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
c.tts("r1", "길게 말하는 중");
|
||||
await until(() => w.received.length >= 2);
|
||||
// The worker dies...
|
||||
for (const s of w.sockets) s.terminate();
|
||||
await new Promise<void>((r) => w.wss.close(() => r()));
|
||||
await until(() => disconnects === 1);
|
||||
expect(done).toEqual([{ id: "r1", chunks: 0, cancelled: false, failed: true }]);
|
||||
expect(c.isReady).toBe(false);
|
||||
// Audio while it is down is dropped, not queued.
|
||||
c.sendAudio("1", Buffer.alloc(4));
|
||||
// ...and comes back on the same port.
|
||||
w = await fakeWorker(port);
|
||||
cleanups.push(() => void w.wss.close());
|
||||
await until(() => readies === 2);
|
||||
expect(w.received[0]).toMatchObject({ type: "hello", token: TOKEN });
|
||||
expect(w.received.filter(Buffer.isBuffer)).toEqual([]);
|
||||
});
|
||||
|
||||
it("settings changed with configure() are sent now and in every later hello", async () => {
|
||||
let w = await fakeWorker();
|
||||
const port = w.port;
|
||||
const c = client(w.url, { stt: { language: "ko" } });
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
c.configure({ partials: false });
|
||||
await until(() => w.received.length >= 2);
|
||||
expect(w.received[1]).toEqual({ type: "config", partials: false });
|
||||
for (const s of w.sockets) s.terminate();
|
||||
await new Promise<void>((r) => w.wss.close(() => r()));
|
||||
await until(() => !c.isReady);
|
||||
w = await fakeWorker(port);
|
||||
cleanups.push(() => void w.wss.close());
|
||||
await until(() => c.isReady);
|
||||
expect(w.received[0]).toMatchObject({ stt: { language: "ko", partials: false } });
|
||||
});
|
||||
|
||||
it("tts while disconnected fails at once instead of waiting forever", async () => {
|
||||
const c = client("ws://127.0.0.1:1");
|
||||
const done: TtsDone[] = [];
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
expect(c.tts("r1", "x")).toBe(false);
|
||||
await until(() => done.length === 1);
|
||||
expect(done[0]).toMatchObject({ id: "r1", failed: true });
|
||||
});
|
||||
});
|
||||
@@ -1,85 +0,0 @@
|
||||
import { mkdtempSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { once } from "node:events";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { VoiceWorkerProcess } from "../src/core/workerProcess.js";
|
||||
|
||||
// A stand-in `voice_worker` package run by a real Python, so the spawn line is the one the app uses.
|
||||
const PYTHON = process.env.JARVIS_TEST_PYTHON ?? (process.platform === "win32" ? "python" : "python3");
|
||||
const FAKE_MAIN = `
|
||||
import json, os, sys, time
|
||||
out = os.environ["FAKE_OUT"]
|
||||
runs = len(os.listdir(out))
|
||||
with open(os.path.join(out, f"run{runs}.json"), "w") as f:
|
||||
json.dump({"argv": sys.argv[1:], "token": os.environ.get("JARVIS_VOICE_TOKEN")}, f)
|
||||
if os.environ.get("FAKE_CRASH_FIRST") == "1" and runs == 0:
|
||||
sys.exit(3)
|
||||
print("loading models...", flush=True)
|
||||
print("READY 127.0.0.1:" + sys.argv[sys.argv.index("--port") + 1], flush=True)
|
||||
time.sleep(60)
|
||||
`;
|
||||
|
||||
function fakeWorkerDir() {
|
||||
const root = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-fake-worker-"));
|
||||
mkdirSync(join(root, "voice_worker"));
|
||||
writeFileSync(join(root, "voice_worker", "__init__.py"), "");
|
||||
writeFileSync(join(root, "voice_worker", "__main__.py"), FAKE_MAIN);
|
||||
const out = join(root, "out");
|
||||
mkdirSync(out);
|
||||
return { root, out, run: (n: number) => JSON.parse(readFileSync(join(out, `run${n}.json`), "utf8")) };
|
||||
}
|
||||
|
||||
const procs: VoiceWorkerProcess[] = [];
|
||||
afterEach(async () => {
|
||||
for (const p of procs.splice(0)) await p.stop(1000);
|
||||
});
|
||||
|
||||
describe("VoiceWorkerProcess", () => {
|
||||
it("runs python -m voice_worker on loopback with the token only in the environment", async () => {
|
||||
const d = fakeWorkerDir();
|
||||
const p = new VoiceWorkerProcess({ python: PYTHON, cwd: d.root, args: ["--melo-python", "/x/python"],
|
||||
env: { FAKE_OUT: d.out } });
|
||||
procs.push(p);
|
||||
const ready = once(p, "ready");
|
||||
await p.start();
|
||||
await ready;
|
||||
const run = d.run(0);
|
||||
const port = new URL(p.url).port;
|
||||
expect(run.argv).toEqual(["--host", "127.0.0.1", "--port", port, "--melo-python", "/x/python"]);
|
||||
expect(run.token).toBe(p.token);
|
||||
expect(p.token.length).toBeGreaterThanOrEqual(40);
|
||||
expect(run.argv.join(" ")).not.toContain(p.token);
|
||||
expect(new URL(p.url).hostname).toBe("127.0.0.1");
|
||||
});
|
||||
|
||||
it("restarts a worker that dies, on the same port with the same token", async () => {
|
||||
const d = fakeWorkerDir();
|
||||
const p = new VoiceWorkerProcess({ python: PYTHON, cwd: d.root, args: [], restartDelayMs: 50,
|
||||
env: { FAKE_OUT: d.out, FAKE_CRASH_FIRST: "1" } });
|
||||
procs.push(p);
|
||||
const exited = once(p, "exit");
|
||||
const ready = once(p, "ready");
|
||||
await p.start();
|
||||
expect((await exited)[0]).toMatchObject({ code: 3 });
|
||||
await ready;
|
||||
expect(p.restarts).toBe(1);
|
||||
expect(d.run(1).argv).toEqual(d.run(0).argv);
|
||||
expect(d.run(1).token).toBe(d.run(0).token);
|
||||
});
|
||||
|
||||
it("stop ends the worker for good", async () => {
|
||||
const d = fakeWorkerDir();
|
||||
const p = new VoiceWorkerProcess({ python: PYTHON, cwd: d.root, args: [], restartDelayMs: 20,
|
||||
env: { FAKE_OUT: d.out } });
|
||||
const ready = once(p, "ready");
|
||||
await p.start();
|
||||
await ready;
|
||||
let exits = 0;
|
||||
p.on("exit", () => exits++);
|
||||
await p.stop(2000);
|
||||
await new Promise((r) => setTimeout(r, 200));
|
||||
expect(exits).toBe(1);
|
||||
expect(p.restarts).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -1,89 +0,0 @@
|
||||
/**
|
||||
* The app's worker client against the real Python voice worker server (with fake engines), so both
|
||||
* sides of the protocol are checked together. Needs a Python with the worker's requirements:
|
||||
* JARVIS_VOICE_PYTHON=<venv python> npm test
|
||||
*/
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { createInterface } from "node:readline";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { Playback } from "../src/core/playback.js";
|
||||
import { VoiceWorkerClient, type Final, type TtsDone } from "../src/core/workerClient.js";
|
||||
import { freePort } from "../src/core/workerProcess.js";
|
||||
|
||||
const PYTHON = process.env.JARVIS_VOICE_PYTHON;
|
||||
const SRC = fileURLToPath(new URL("../../src", import.meta.url));
|
||||
const FIXTURE = fileURLToPath(new URL("./fixtures/fake_engine_worker.py", import.meta.url));
|
||||
const TOKEN = "protocol-test-token";
|
||||
const FMT = { rate: 48000, channels: 2 } as const;
|
||||
|
||||
/** 20 ms of a 48 kHz stereo tone whose amplitude and pitch (x1000 Hz) are `level`. */
|
||||
function frame(level: number): Buffer {
|
||||
const n = 960;
|
||||
const buf = Buffer.alloc(n * 4);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const v = Math.round(level * Math.sin((2 * Math.PI * level * 1000 * i) / 48000) * 32767);
|
||||
buf.writeInt16LE(v, i * 4);
|
||||
buf.writeInt16LE(v, i * 4 + 2);
|
||||
}
|
||||
return buf;
|
||||
}
|
||||
|
||||
describe.skipIf(!PYTHON)("worker protocol against the Python server", () => {
|
||||
let child: ChildProcess;
|
||||
let client: VoiceWorkerClient;
|
||||
|
||||
beforeAll(async () => {
|
||||
const port = await freePort();
|
||||
child = spawn(PYTHON!, [FIXTURE, String(port)], {
|
||||
env: { ...process.env, PYTHONPATH: SRC, JARVIS_VOICE_TOKEN: TOKEN },
|
||||
stdio: ["ignore", "pipe", "inherit"],
|
||||
});
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
child.once("exit", (c) => reject(new Error(`fake worker exited ${c}`)));
|
||||
createInterface({ input: child.stdout! }).on("line", (l) => l.startsWith("READY") && resolve());
|
||||
});
|
||||
client = new VoiceWorkerClient({ url: `ws://127.0.0.1:${port}`, token: TOKEN, audio: FMT, tts: FMT,
|
||||
stt: { partials: false, hotwords: ["자비스"] } });
|
||||
client.start();
|
||||
const ok = await client.waitReady(5000);
|
||||
expect(ok).toMatchObject({ stt_ready: true, tts_ready: true });
|
||||
expect(ok.stt).toMatchObject({ partials: false, hotwords: ["자비스"] });
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
client?.close();
|
||||
child?.kill();
|
||||
});
|
||||
|
||||
it("Discord-format audio from two speakers comes back as each speaker's final", async () => {
|
||||
const finals: Final[] = [];
|
||||
client.on("final", (f) => finals.push(f));
|
||||
for (let i = 0; i < 30; i++) {
|
||||
client.sendAudio("111111111111111111", frame(0.9));
|
||||
client.sendAudio("222222222222222222", frame(0.6));
|
||||
}
|
||||
client.speakingEnd("111111111111111111");
|
||||
// The second speaker leaves mid-utterance: the final still comes.
|
||||
client.speakerLeave("222222222222222222");
|
||||
const end = Date.now() + 5000;
|
||||
while (finals.length < 2 && Date.now() < end) await new Promise((r) => setTimeout(r, 20));
|
||||
const bySpeaker = Object.fromEntries(finals.map((f) => [f.speaker, f]));
|
||||
expect(bySpeaker["111111111111111111"]).toMatchObject({ text: "level 0.9", end_reason: "speaking_end", utt: 1 });
|
||||
expect(bySpeaker["222222222222222222"]).toMatchObject({ text: "level 0.6", end_reason: "speaking_end", utt: 1 });
|
||||
});
|
||||
|
||||
it("TTS sentences play in order through Playback and the request ends cleanly", async () => {
|
||||
const played: Buffer[] = [];
|
||||
const playback = new Playback(client, { play: (pcm) => { played.push(pcm); setTimeout(() => playback.clipEnded(), 5); }, stop: () => {} });
|
||||
client.on("tts_audio", (f) => playback.audio(f.id, f.seq, f.pcm));
|
||||
client.on("tts_done", (d: TtsDone) => playback.done(d));
|
||||
const ended = new Promise((resolve) => playback.once("end", resolve));
|
||||
const text = "첫 번째 문장입니다. 두 번째 문장은 조금 더 깁니다.";
|
||||
playback.say(text);
|
||||
expect(await ended).toMatchObject({ played: 2, cancelled: false, failed: false });
|
||||
// 10 samples per character at 48 kHz, mono duplicated to stereo, 2 bytes each.
|
||||
const sentences = ["첫 번째 문장입니다.", "두 번째 문장은 조금 더 깁니다."];
|
||||
expect(played.map((p) => p.length)).toEqual(sentences.map((s) => s.length * 10 * 2 * 2));
|
||||
});
|
||||
});
|
||||
359
app/tests/live_discord.rs
Normal file
359
app/tests/live_discord.rs
Normal file
@@ -0,0 +1,359 @@
|
||||
//! The Discord voice core end to end on real Discord, with the real voice worker on the GPU.
|
||||
//!
|
||||
//! Two bots in one voice channel: the "Jarvis" bot runs the core; the "speaker" bot plays recorded
|
||||
//! Korean speech into the channel, records what Jarvis says, and writes in the channel's chat.
|
||||
//! Opt-in; it refuses to run if a person is in the channel or either bot is already in voice.
|
||||
//!
|
||||
//! JARVIS_LIVE_DISCORD=1 JARVIS_LIVE_BOT_TOKEN=... JARVIS_LIVE_SPEAKER_TOKEN=... \
|
||||
//! JARVIS_LIVE_GUILD=<server id> JARVIS_LIVE_VOICE_CHANNEL=<voice channel id> \
|
||||
//! JARVIS_VOICE_PYTHON=<worker venv python> JARVIS_MELO_PYTHON=<MeloTTS venv python> \
|
||||
//! cargo test --release --test live_discord -- --nocapture
|
||||
//!
|
||||
//! On Linux, put the pip CUDA libraries on LD_LIBRARY_PATH (nvidia/cublas/lib, nvidia/cudnn/lib).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use jarvis_app::discord::{CoreEvent, DiscordCore};
|
||||
use jarvis_app::receive::{opus_payload, OpusFrames, SpeakerStream};
|
||||
use jarvis_app::settings::parse_settings;
|
||||
use jarvis_app::worker_client::{WorkerClient, WorkerClientOptions, WorkerEvent};
|
||||
use jarvis_app::worker_process::{ProcessEvent, WorkerProcess, WorkerProcessOptions};
|
||||
use serde_json::{json, Value};
|
||||
use serenity::all::{ChannelId, Client, Context, CreateMessage, EventHandler, GatewayIntents, GuildId, Http, Ready, UserId};
|
||||
use songbird::driver::DecodeMode;
|
||||
use songbird::input::Input;
|
||||
use songbird::{Config, CoreEvent as VoiceEv, Event, EventContext, EventHandler as VoiceHandler, SerenityInit, Songbird, TrackEvent};
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
const COMMAND: &str = "jarvis-live-test";
|
||||
|
||||
fn env(k: &str) -> String {
|
||||
std::env::var(k).unwrap_or_else(|_| panic!("{k} is not set"))
|
||||
}
|
||||
|
||||
fn cer(reference: &str, hyp: &str) -> f64 {
|
||||
let norm = |s: &str| s.chars().filter(|c| c.is_alphanumeric()).collect::<Vec<_>>();
|
||||
let (a, b) = (norm(reference), norm(hyp));
|
||||
let mut prev: Vec<usize> = (0..=b.len()).collect();
|
||||
for i in 1..=a.len() {
|
||||
let mut cur = vec![i];
|
||||
for j in 1..=b.len() {
|
||||
cur.push((prev[j] + 1).min(cur[j - 1] + 1).min(prev[j - 1] + usize::from(a[i - 1] != b[j - 1])));
|
||||
}
|
||||
prev = cur;
|
||||
}
|
||||
prev[b.len()] as f64 / a.len().max(1) as f64
|
||||
}
|
||||
|
||||
/// Seconds into a 16 kHz mono PCM16 WAV where the last sound above -34 dBFS ends.
|
||||
fn speech_end_seconds(wav: &[u8]) -> f64 {
|
||||
let data = &wav[44..];
|
||||
let n = data.len() / 2;
|
||||
(0..n).rev().find(|&k| i16::from_le_bytes([data[2 * k], data[2 * k + 1]]).unsigned_abs() > 655)
|
||||
.map_or(0.0, |k| (k + 1) as f64 / 16000.0)
|
||||
}
|
||||
|
||||
struct ReadyOnce(mpsc::UnboundedSender<Ready>);
|
||||
#[async_trait]
|
||||
impl EventHandler for ReadyOnce {
|
||||
async fn ready(&self, _: Context, r: Ready) {
|
||||
let _ = self.0.send(r);
|
||||
}
|
||||
}
|
||||
|
||||
/// Jarvis's voice as the speaker bot hears it, decoded the same way the core decodes (songbird's
|
||||
/// own decoder garbles DAVE audio, see src/receive.rs).
|
||||
#[derive(Clone, Default)]
|
||||
struct Ears {
|
||||
jarvis: u64,
|
||||
ssrc: Arc<Mutex<HashMap<u32, u64>>>,
|
||||
stream: Arc<Mutex<Option<SpeakerStream<OpusFrames>>>>,
|
||||
pcm: Arc<Mutex<Vec<i16>>>,
|
||||
/// When the last audible frame arrived.
|
||||
last_packet: Arc<Mutex<Option<Instant>>>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl VoiceHandler for Ears {
|
||||
async fn act(&self, ctx: &EventContext<'_>) -> Option<Event> {
|
||||
match ctx {
|
||||
EventContext::SpeakingStateUpdate(s) => {
|
||||
if let Some(u) = s.user_id {
|
||||
self.ssrc.lock().unwrap().insert(s.ssrc, u.0);
|
||||
}
|
||||
}
|
||||
EventContext::RtpPacket(p) => {
|
||||
let (seq, ssrc, opus) = opus_payload(&p.packet, p.payload_offset, p.payload_end_pad)?;
|
||||
if self.ssrc.lock().unwrap().get(&ssrc) != Some(&self.jarvis) {
|
||||
return None;
|
||||
}
|
||||
let mut st = self.stream.lock().unwrap();
|
||||
let frames = st.get_or_insert_with(|| SpeakerStream::new(OpusFrames::new())).push(seq, opus.to_vec());
|
||||
let mut pcm = self.pcm.lock().unwrap();
|
||||
for f in frames {
|
||||
// Audible: louder than -36 dBFS. The sender's closing silence frames do not count.
|
||||
let rms = (f.iter().map(|&x| f64::from(x).powi(2)).sum::<f64>() / f.len().max(1) as f64).sqrt();
|
||||
if rms > 500.0 {
|
||||
*self.last_packet.lock().unwrap() = Some(Instant::now());
|
||||
}
|
||||
pcm.extend_from_slice(&f);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
struct Ended(mpsc::UnboundedSender<()>);
|
||||
#[async_trait]
|
||||
impl VoiceHandler for Ended {
|
||||
async fn act(&self, _: &EventContext<'_>) -> Option<Event> {
|
||||
let _ = self.0.send(());
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
async fn until(what: &str, ms: u64, mut pred: impl FnMut() -> bool) {
|
||||
let deadline = Instant::now() + Duration::from_millis(ms);
|
||||
while !pred() {
|
||||
assert!(Instant::now() < deadline, "timed out waiting for {what}");
|
||||
tokio::time::sleep(Duration::from_millis(20)).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn discord_voice_core_on_real_discord() {
|
||||
if std::env::var("JARVIS_LIVE_DISCORD").as_deref() != Ok("1") {
|
||||
eprintln!("skipped: set JARVIS_LIVE_DISCORD=1");
|
||||
return;
|
||||
}
|
||||
let guild: u64 = env("JARVIS_LIVE_GUILD").parse().unwrap();
|
||||
let voice: u64 = env("JARVIS_LIVE_VOICE_CHANNEL").parse().unwrap();
|
||||
let bot_token = env("JARVIS_LIVE_BOT_TOKEN");
|
||||
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
|
||||
let samples = root.join("../scripts/m0/assets/ko_samples");
|
||||
|
||||
// The speaker bot first: it checks the channel is free before anything joins.
|
||||
let speaker_songbird = Songbird::serenity_from_config(Config::default().decode_mode(DecodeMode::Decrypt));
|
||||
let (rtx, mut rrx) = mpsc::unbounded_channel();
|
||||
let mut speaker = Client::builder(env("JARVIS_LIVE_SPEAKER_TOKEN"),
|
||||
GatewayIntents::GUILDS | GatewayIntents::GUILD_VOICE_STATES | GatewayIntents::GUILD_MESSAGES | GatewayIntents::MESSAGE_CONTENT)
|
||||
.event_handler(ReadyOnce(rtx)).register_songbird_with(speaker_songbird.clone()).await.unwrap();
|
||||
let (s_http, s_cache, s_shards) = (speaker.http.clone(), speaker.cache.clone(), speaker.shard_manager.clone());
|
||||
tokio::spawn(async move { speaker.start().await.unwrap() });
|
||||
let speaker_id = rrx.recv().await.unwrap().user.id.get();
|
||||
let jarvis_app_id = Http::new(&bot_token).get_current_application_info().await.unwrap().id.get();
|
||||
until("guild in cache", 10_000, || s_cache.guild(GuildId::new(guild)).is_some()).await;
|
||||
{
|
||||
let g = s_cache.guild(GuildId::new(guild)).unwrap();
|
||||
let people = g.voice_states.values().filter(|v| v.channel_id.map(|c| c.get()) == Some(voice)
|
||||
&& !s_cache.user(v.user_id).map(|u| u.bot).unwrap_or(false)).count();
|
||||
assert_eq!(people, 0, "refusing to test: people are in the voice channel");
|
||||
for id in [speaker_id, jarvis_app_id] {
|
||||
assert!(g.voice_states.get(&UserId::new(id)).and_then(|v| v.channel_id).is_none(),
|
||||
"refusing to test: bot {id} is in a voice channel");
|
||||
}
|
||||
}
|
||||
|
||||
// The real voice worker.
|
||||
let mut args = vec![];
|
||||
if let Ok(m) = std::env::var("JARVIS_MELO_PYTHON") {
|
||||
args = vec!["--melo-python".to_string(), m];
|
||||
}
|
||||
let wp = WorkerProcess::new(WorkerProcessOptions::new(env("JARVIS_VOICE_PYTHON"), root.join("../src"), args)).unwrap();
|
||||
let mut pev = wp.subscribe();
|
||||
wp.start();
|
||||
tokio::time::timeout(Duration::from_secs(240), async {
|
||||
while pev.recv().await.unwrap() != ProcessEvent::Ready {}
|
||||
}).await.expect("worker ready");
|
||||
let mut wo = WorkerClientOptions::new(wp.url(), wp.token.clone());
|
||||
wo.stt = json!({"hotwords": ["자비스"]}).as_object().unwrap().clone();
|
||||
let (worker, worker_events) = WorkerClient::new(wo);
|
||||
worker.start();
|
||||
worker.wait_ready(Duration::from_secs(10)).await.expect("worker hello");
|
||||
|
||||
let settings = parse_settings(&json!({
|
||||
"discord": {"guildId": guild.to_string(), "voiceChannelId": voice.to_string(), "commandName": COMMAND},
|
||||
// The "person" here is a bot, so bots must be heard.
|
||||
"voice": {"listenToBots": true},
|
||||
// The voice channel's own chat stands in for a registered chat channel.
|
||||
"chatChannels": [{"id": voice.to_string(), "name": "테스트", "description": "라이브 테스트"}],
|
||||
})).unwrap();
|
||||
let (core, mut core_rx) = DiscordCore::new(settings, bot_token.clone(), worker.clone(), worker_events);
|
||||
let events: Arc<Mutex<Vec<CoreEvent>>> = Arc::default();
|
||||
let ev2 = events.clone();
|
||||
tokio::spawn(async move {
|
||||
while let Some(e) = core_rx.recv().await {
|
||||
ev2.lock().unwrap().push(e);
|
||||
}
|
||||
});
|
||||
core.start().await.expect("core start"); // logs in, registers the command, auto-joins, posts the notice
|
||||
assert_eq!(core.voice_channel_id(), Some(voice));
|
||||
let jarvis = core.bot_id().unwrap();
|
||||
|
||||
let call = speaker_songbird.join(GuildId::new(guild), ChannelId::new(voice)).await.expect("speaker joins");
|
||||
let ears = Ears { jarvis, ..Default::default() };
|
||||
{
|
||||
let mut c = call.lock().await;
|
||||
c.add_global_event(VoiceEv::SpeakingStateUpdate.into(), ears.clone());
|
||||
c.add_global_event(VoiceEv::RtpPacket.into(), ears.clone());
|
||||
}
|
||||
let mut report = serde_json::Map::new();
|
||||
let steps = futures_util::FutureExt::catch_unwind(std::panic::AssertUnwindSafe(async {
|
||||
|
||||
// 1. The slash command is registered.
|
||||
let jarvis_http = Http::new(&bot_token);
|
||||
jarvis_http.set_application_id(jarvis_app_id.into());
|
||||
let cmds = jarvis_http.get_guild_commands(GuildId::new(guild)).await.unwrap();
|
||||
let cmd = cmds.iter().find(|c| c.name == COMMAND).expect("command registered");
|
||||
let mut subs: Vec<_> = cmd.options.iter().map(|o| o.name.clone()).collect();
|
||||
subs.sort();
|
||||
assert_eq!(subs, ["join", "leave", "stop"]);
|
||||
|
||||
// 2. Recorded sentences played into the channel come back as the speaker's finals.
|
||||
let refs: Value = serde_json::from_str(&std::fs::read_to_string(samples.join("refs.json")).unwrap()).unwrap();
|
||||
let mut stt = vec![];
|
||||
let mut total = 0.0;
|
||||
for r in refs["samples"].as_array().unwrap().iter().take(5) {
|
||||
let wav = std::fs::read(samples.join(r["file"].as_str().unwrap())).unwrap();
|
||||
let reference = r["text"].as_str().unwrap();
|
||||
let before = events.lock().unwrap().len();
|
||||
let (etx, mut erx) = mpsc::unbounded_channel();
|
||||
let track = call.lock().await.play_only_input(Input::from(wav.clone()));
|
||||
track.add_event(Event::Track(TrackEvent::End), Ended(etx)).unwrap();
|
||||
let deadline = Instant::now() + Duration::from_secs(5);
|
||||
let started = loop {
|
||||
if track.get_info().await.is_ok_and(|i| i.position > Duration::ZERO) {
|
||||
break Instant::now();
|
||||
}
|
||||
assert!(Instant::now() < deadline, "timed out waiting for the track to play");
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
};
|
||||
let speech_end = started + Duration::from_secs_f64(speech_end_seconds(&wav));
|
||||
let mut final_at = None;
|
||||
until("final", 20_000, || {
|
||||
let got = events.lock().unwrap()[before..].iter().any(|e| matches!(e, CoreEvent::Final(f) if f.heard.speaker_id == speaker_id));
|
||||
if got && final_at.is_none() {
|
||||
final_at = Some(Instant::now());
|
||||
}
|
||||
got
|
||||
}).await;
|
||||
tokio::time::timeout(Duration::from_secs(20), erx.recv()).await.expect("track end");
|
||||
tokio::time::sleep(Duration::from_millis(800)).await;
|
||||
let mine: Vec<_> = events.lock().unwrap()[before..].iter().filter_map(|e| match e {
|
||||
CoreEvent::Final(f) if f.heard.speaker_id == speaker_id => Some(f.clone()),
|
||||
_ => None,
|
||||
}).collect();
|
||||
let hyp = mine.iter().map(|f| f.text.as_str()).collect::<Vec<_>>().join(" ");
|
||||
assert!(!mine[0].heard.speaker_name.is_empty());
|
||||
let c = cer(reference, &hyp);
|
||||
total += c;
|
||||
let latency = final_at.unwrap().saturating_duration_since(speech_end).as_millis();
|
||||
stt.push(json!({"ref": reference, "hyp": hyp, "cer": c, "speech_end_to_final_ms": latency,
|
||||
"end_reason": mine.iter().map(|f| f.end_reason.clone()).collect::<Vec<_>>()}));
|
||||
assert!(!hyp.is_empty(), "nothing heard for {reference}");
|
||||
}
|
||||
report.insert("stt".into(), Value::from(stt.clone()));
|
||||
let mean = total / (stt.len() as f64);
|
||||
assert!(mean < 0.15, "mean CER {mean}");
|
||||
|
||||
// 3. Jarvis speaks; the speaker bot hears it and the STT reads it back.
|
||||
let text = "안녕하세요, 저는 자비스입니다. 디스코드 음성 연결을 확인하고 있어요.";
|
||||
ears.pcm.lock().unwrap().clear();
|
||||
let id = core.say(text, false).expect("in voice");
|
||||
until("speech to end", 30_000, || events.lock().unwrap().iter().any(|e| matches!(e, CoreEvent::SpeakEnd(x) if x.id == id))).await;
|
||||
let end = events.lock().unwrap().iter().find_map(|e| match e { CoreEvent::SpeakEnd(x) if x.id == id => Some(x.clone()), _ => None }).unwrap();
|
||||
assert!(!end.cancelled && !end.failed && end.played == 2, "{end:?}");
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
let heard = ears.pcm.lock().unwrap().clone();
|
||||
let seconds = heard.len() as f64 / 96000.0;
|
||||
assert!(seconds > 2.0, "heard only {seconds} s");
|
||||
let mut eo = WorkerClientOptions::new(wp.url(), wp.token.clone());
|
||||
eo.stt = json!({"partials": false}).as_object().unwrap().clone();
|
||||
let (echo, mut echo_rx) = WorkerClient::new(eo);
|
||||
echo.start();
|
||||
echo.wait_ready(Duration::from_secs(5)).await.unwrap();
|
||||
for chunk in heard.chunks(1920) {
|
||||
echo.send_audio("echo", chunk);
|
||||
}
|
||||
for _ in 0..25 {
|
||||
echo.send_audio("echo", &[0; 1920]);
|
||||
}
|
||||
echo.speaking_end("echo");
|
||||
let mut back = vec![];
|
||||
let _ = tokio::time::timeout(Duration::from_secs(3), async {
|
||||
while let Some(e) = echo_rx.recv().await {
|
||||
if let WorkerEvent::Final(f) = e {
|
||||
back.push(f.text);
|
||||
}
|
||||
}
|
||||
}).await;
|
||||
echo.close();
|
||||
let hyp = back.join(" ");
|
||||
report.insert("tts_read_back".into(), json!({"seconds": seconds, "hyp": hyp, "cer": cer(text, &hyp)}));
|
||||
assert!(cer(text, &hyp) < 0.2, "read back: {hyp}");
|
||||
|
||||
// 4. Stop silences it within a few frames and cancels the rest.
|
||||
let long = "이 문장은 정지 명령을 시험하려고 길게 말하는 첫 번째 문장입니다. 두 번째 문장도 꽤 길게 이어집니다. \
|
||||
세 번째 문장까지 들리면 정지가 안 된 것입니다. 네 번째 문장은 절대 들리면 안 됩니다.";
|
||||
let id = core.say(long, false).unwrap();
|
||||
until("speech to start", 30_000, || events.lock().unwrap().iter().any(|e| *e == CoreEvent::SpeakStart(id.clone()))).await;
|
||||
tokio::time::sleep(Duration::from_millis(1500)).await;
|
||||
let stop_at = Instant::now();
|
||||
assert_eq!(core.stop(), 1);
|
||||
tokio::time::sleep(Duration::from_millis(1500)).await;
|
||||
let ends: Vec<_> = events.lock().unwrap().iter().filter_map(|e| match e { CoreEvent::SpeakEnd(x) if x.id == id => Some(x.clone()), _ => None }).collect();
|
||||
assert!(ends.len() == 1 && ends[0].cancelled, "{ends:?}");
|
||||
let tail = ears.last_packet.lock().unwrap().map(|t| t.saturating_duration_since(stop_at).as_millis() as i64).unwrap_or(-1);
|
||||
report.insert("last_sound_after_stop_ms".into(), json!(tail));
|
||||
assert!(tail < 400, "audio went on {tail} ms after stop");
|
||||
assert!(!core.speaking());
|
||||
|
||||
// 5. Chat: live event, history with the join notice, and a send that pings nobody.
|
||||
let marker = format!("라이브 테스트 메시지 {}", stop_at.elapsed().as_nanos());
|
||||
ChannelId::new(voice).send_message(&s_http, CreateMessage::new().content(&marker)).await.unwrap();
|
||||
until("chat event", 10_000, || events.lock().unwrap().iter().any(|e| matches!(e, CoreEvent::Chat(c) if c.content == marker))).await;
|
||||
let ev = events.lock().unwrap().iter().find_map(|e| match e { CoreEvent::Chat(c) if c.content == marker => Some(c.clone()), _ => None }).unwrap();
|
||||
assert!(ev.channel_id == voice && ev.author_id == speaker_id && ev.author_is_bot && !ev.from_self);
|
||||
let history = core.read_chat(voice, 20).await.unwrap();
|
||||
assert!(history.iter().any(|m| m.content == marker && !m.from_self));
|
||||
assert!(history.iter().any(|m| m.from_self && m.content.contains("받아 적습니다")), "join notice missing");
|
||||
let reply = format!("자비스 답장 @everyone {}", stop_at.elapsed().as_nanos());
|
||||
let ids = core.send_chat(voice, &reply).await.unwrap();
|
||||
let sent = ChannelId::new(voice).message(&s_http, serenity::all::MessageId::new(ids[0])).await.unwrap();
|
||||
assert_eq!(sent.content, reply);
|
||||
assert!(!sent.mention_everyone);
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
assert!(!events.lock().unwrap().iter().any(|e| matches!(e, CoreEvent::Chat(c) if c.content == reply)));
|
||||
|
||||
// 6. Leaving.
|
||||
assert!(core.leave("command").await);
|
||||
until("Jarvis to leave", 10_000, || {
|
||||
s_cache.guild(GuildId::new(guild)).and_then(|g| g.voice_states.get(&UserId::new(jarvis)).and_then(|v| v.channel_id)).is_none()
|
||||
}).await;
|
||||
assert_eq!(core.voice_channel_id(), None);
|
||||
assert!(core.say("아무도 못 듣는 말", false).is_none());
|
||||
|
||||
println!("live report {}", serde_json::to_string_pretty(&Value::Object(report.clone())).unwrap());
|
||||
})).await;
|
||||
|
||||
// Clean up even when a step failed: the test command, the speaker, the core, the worker.
|
||||
let http = Http::new(&bot_token);
|
||||
http.set_application_id(jarvis_app_id.into());
|
||||
for c in http.get_guild_commands(GuildId::new(guild)).await.unwrap().iter().filter(|c| c.name == COMMAND) {
|
||||
let _ = http.delete_guild_command(GuildId::new(guild), c.id).await;
|
||||
}
|
||||
let _ = speaker_songbird.remove(GuildId::new(guild)).await;
|
||||
s_shards.shutdown_all().await;
|
||||
core.close().await;
|
||||
worker.close();
|
||||
wp.stop().await;
|
||||
if let Err(panic) = steps {
|
||||
std::panic::resume_unwind(panic);
|
||||
}
|
||||
}
|
||||
230
app/tests/worker_client.rs
Normal file
230
app/tests/worker_client.rs
Normal file
@@ -0,0 +1,230 @@
|
||||
//! The worker client against a stand-in worker server (hello, events, reconnect, failed TTS).
|
||||
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use futures_util::{SinkExt, StreamExt};
|
||||
use jarvis_app::frames::{encode_audio_frame, KIND_TTS};
|
||||
use jarvis_app::playback::TtsBackend;
|
||||
use jarvis_app::worker_client::{TtsDone, WorkerClient, WorkerClientOptions, WorkerEvent};
|
||||
use serde_json::{json, Value};
|
||||
use tokio::net::TcpListener;
|
||||
use tokio::sync::mpsc;
|
||||
use tokio_tungstenite::tungstenite::Message;
|
||||
|
||||
const TOKEN: &str = "t0ken";
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
enum Got {
|
||||
Json(Value),
|
||||
Bin(Vec<u8>),
|
||||
}
|
||||
|
||||
/// A fake worker: checks the hello, records what arrives, and lets the test push frames.
|
||||
struct FakeWorker {
|
||||
port: u16,
|
||||
received: Arc<Mutex<Vec<Got>>>,
|
||||
push: Arc<Mutex<Option<mpsc::UnboundedSender<Message>>>>,
|
||||
task: tokio::task::JoinHandle<()>,
|
||||
conns: Arc<Mutex<Vec<tokio::task::JoinHandle<()>>>>,
|
||||
}
|
||||
|
||||
impl FakeWorker {
|
||||
async fn start(port: u16) -> Self {
|
||||
let listener = TcpListener::bind(("127.0.0.1", port)).await.unwrap();
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
let received = Arc::new(Mutex::new(vec![]));
|
||||
let push = Arc::new(Mutex::new(None));
|
||||
let conns = Arc::new(Mutex::new(vec![]));
|
||||
let (r, p, cs) = (received.clone(), push.clone(), conns.clone());
|
||||
let task = tokio::spawn(async move {
|
||||
while let Ok((tcp, _)) = listener.accept().await {
|
||||
let ws = tokio_tungstenite::accept_async(tcp).await.unwrap();
|
||||
let (mut tx, mut rx) = ws.split();
|
||||
let (ptx, mut prx) = mpsc::unbounded_channel::<Message>();
|
||||
*p.lock().unwrap() = Some(ptx.clone());
|
||||
let r = r.clone();
|
||||
cs.lock().unwrap().push(tokio::spawn(async move {
|
||||
loop {
|
||||
tokio::select! {
|
||||
m = prx.recv() => match m { Some(m) => { if tx.send(m).await.is_err() { break } } None => break },
|
||||
m = rx.next() => match m {
|
||||
Some(Ok(Message::Text(t))) => {
|
||||
let v: Value = serde_json::from_str(t.as_str()).unwrap();
|
||||
if v["type"] == "hello" {
|
||||
if v["token"] != TOKEN { let _ = tx.close().await; break; }
|
||||
let ok = json!({"type": "hello_ok", "stt_ready": true, "tts_ready": true, "stt": v["stt"]});
|
||||
let _ = ptx.send(Message::text(ok.to_string()));
|
||||
}
|
||||
r.lock().unwrap().push(Got::Json(v));
|
||||
}
|
||||
Some(Ok(Message::Binary(b))) => r.lock().unwrap().push(Got::Bin(b.to_vec())),
|
||||
Some(Ok(_)) => {}
|
||||
_ => break,
|
||||
},
|
||||
}
|
||||
}
|
||||
}));
|
||||
}
|
||||
});
|
||||
Self { port, received, push, task, conns }
|
||||
}
|
||||
|
||||
fn url(&self) -> String {
|
||||
format!("ws://127.0.0.1:{}", self.port)
|
||||
}
|
||||
|
||||
fn got(&self) -> Vec<Got> {
|
||||
self.received.lock().unwrap().clone()
|
||||
}
|
||||
|
||||
fn send(&self, m: Message) {
|
||||
self.push.lock().unwrap().as_ref().unwrap().send(m).unwrap();
|
||||
}
|
||||
|
||||
/// The worker dies: stop listening and drop the connection.
|
||||
fn kill(self) -> u16 {
|
||||
self.task.abort();
|
||||
for c in self.conns.lock().unwrap().drain(..) {
|
||||
c.abort();
|
||||
}
|
||||
*self.push.lock().unwrap() = None;
|
||||
self.port
|
||||
}
|
||||
}
|
||||
|
||||
fn client(url: &str, stt: Value) -> (WorkerClient, mpsc::UnboundedReceiver<WorkerEvent>) {
|
||||
let mut o = WorkerClientOptions::new(url, TOKEN);
|
||||
o.stt = stt.as_object().cloned().unwrap_or_default();
|
||||
o.backoff = Duration::from_millis(20);
|
||||
o.max_backoff = Duration::from_millis(50);
|
||||
WorkerClient::new(o)
|
||||
}
|
||||
|
||||
async fn until(mut pred: impl FnMut() -> bool) {
|
||||
tokio::time::timeout(Duration::from_secs(3), async {
|
||||
while !pred() {
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("timed out");
|
||||
}
|
||||
|
||||
async fn next_matching(rx: &mut mpsc::UnboundedReceiver<WorkerEvent>, f: impl Fn(&WorkerEvent) -> bool) -> WorkerEvent {
|
||||
tokio::time::timeout(Duration::from_secs(3), async {
|
||||
loop {
|
||||
let e = rx.recv().await.expect("events closed");
|
||||
if f(&e) {
|
||||
return e;
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("timed out")
|
||||
}
|
||||
|
||||
fn tts_frame(id: &str, seq: u32) -> Message {
|
||||
let mut f = vec![KIND_TTS, id.len() as u8];
|
||||
f.extend_from_slice(id.as_bytes());
|
||||
f.extend_from_slice(&seq.to_le_bytes());
|
||||
f.extend_from_slice(&[0; 8]);
|
||||
Message::binary(f)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn hello_carries_token_formats_and_stt_then_audio_flows() {
|
||||
let w = FakeWorker::start(0).await;
|
||||
let (c, _rx) = client(&w.url(), json!({"hotwords": ["자비스"]}));
|
||||
c.start();
|
||||
let ok = c.wait_ready(Duration::from_secs(2)).await.expect("ready");
|
||||
assert_eq!(Value::Object(ok.stt), json!({"hotwords": ["자비스"]}));
|
||||
let pcm = vec![1i16; 1920];
|
||||
c.send_audio("123", &pcm);
|
||||
c.speaking_end("123");
|
||||
c.speaker_leave("123");
|
||||
until(|| w.got().len() >= 4).await;
|
||||
let got = w.got();
|
||||
assert_eq!(got[0], Got::Json(json!({"type": "hello", "token": TOKEN, "audio": {"rate": 48000, "channels": 2},
|
||||
"tts": {"rate": 48000, "channels": 2}, "stt": {"hotwords": ["자비스"]}})));
|
||||
assert_eq!(got[1], Got::Bin(encode_audio_frame("123", &pcm).unwrap()));
|
||||
assert_eq!(got[2..], [Got::Json(json!({"type": "speaking_end", "speaker": "123"})),
|
||||
Got::Json(json!({"type": "speaker_leave", "speaker": "123"}))]);
|
||||
c.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn events_pass_on_and_only_tts_audio_of_open_requests() {
|
||||
let w = FakeWorker::start(0).await;
|
||||
let (c, mut rx) = client(&w.url(), json!({}));
|
||||
c.start();
|
||||
c.wait_ready(Duration::from_secs(2)).await.unwrap();
|
||||
assert!(c.tts("r1", "안녕하세요.", false));
|
||||
until(|| w.got().iter().any(|g| matches!(g, Got::Json(v) if v["type"] == "tts"))).await;
|
||||
w.send(tts_frame("zz", 0)); // not ours
|
||||
w.send(tts_frame("r1", 0));
|
||||
w.send(Message::text(json!({"type": "final", "speaker": "1", "utt": 1, "text": "네", "end_reason": "silence",
|
||||
"audio_ms": 900, "wait_ms": 100, "stt_ms": 80}).to_string()));
|
||||
w.send(Message::text(json!({"type": "tts_done", "id": "r1", "chunks": 1, "cancelled": false, "failed": false}).to_string()));
|
||||
let mut seen = vec![];
|
||||
while seen.len() < 4 {
|
||||
let e = next_matching(&mut rx, |_| true).await;
|
||||
if !matches!(e, WorkerEvent::Ready(_)) {
|
||||
seen.push(e);
|
||||
}
|
||||
if matches!(seen.last(), Some(WorkerEvent::TtsDone(_))) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert!(matches!(&seen[0], WorkerEvent::TtsAudio { id, seq: 0, .. } if id == "r1"));
|
||||
assert!(matches!(&seen[1], WorkerEvent::Final(f) if f.text == "네" && f.end_reason == "silence"));
|
||||
assert_eq!(seen[2], WorkerEvent::TtsDone(TtsDone { id: "r1".into(), chunks: 1, cancelled: false, failed: false }));
|
||||
c.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_refused_tts_request_ends_as_failed() {
|
||||
let w = FakeWorker::start(0).await;
|
||||
let (c, mut rx) = client(&w.url(), json!({}));
|
||||
c.start();
|
||||
c.wait_ready(Duration::from_secs(2)).await.unwrap();
|
||||
c.tts("r9", "x", false);
|
||||
until(|| w.got().len() >= 2).await;
|
||||
w.send(Message::text(json!({"type": "error", "code": "tts_unavailable", "message": "no", "id": "r9"}).to_string()));
|
||||
let e = next_matching(&mut rx, |e| matches!(e, WorkerEvent::TtsDone(_))).await;
|
||||
assert_eq!(e, WorkerEvent::TtsDone(TtsDone { id: "r9".into(), chunks: 0, cancelled: false, failed: true }));
|
||||
c.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reconnects_after_a_restart_and_fails_the_tts_in_flight() {
|
||||
let w = FakeWorker::start(0).await;
|
||||
let (c, mut rx) = client(&w.url(), json!({"language": "ko"}));
|
||||
c.start();
|
||||
c.wait_ready(Duration::from_secs(2)).await.unwrap();
|
||||
c.tts("r1", "길게 말하는 중", false);
|
||||
c.configure(json!({"partials": false}).as_object().unwrap().clone());
|
||||
until(|| w.got().len() >= 3).await;
|
||||
assert_eq!(w.got()[2], Got::Json(json!({"type": "config", "partials": false})));
|
||||
let port = w.kill();
|
||||
let e = next_matching(&mut rx, |e| matches!(e, WorkerEvent::TtsDone(_))).await;
|
||||
assert_eq!(e, WorkerEvent::TtsDone(TtsDone { id: "r1".into(), chunks: 0, cancelled: false, failed: true }));
|
||||
next_matching(&mut rx, |e| *e == WorkerEvent::Disconnected).await;
|
||||
assert!(!c.is_ready());
|
||||
c.send_audio("1", &[0, 0]); // dropped, not queued
|
||||
let w = FakeWorker::start(port).await;
|
||||
next_matching(&mut rx, |e| matches!(e, WorkerEvent::Ready(_))).await;
|
||||
let got = w.got();
|
||||
// The new hello carries the settings changed since, and no stale audio was queued.
|
||||
assert!(matches!(&got[0], Got::Json(v) if v["stt"] == json!({"language": "ko", "partials": false})));
|
||||
assert!(!got.iter().any(|g| matches!(g, Got::Bin(_))));
|
||||
c.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tts_while_disconnected_fails_at_once() {
|
||||
let (c, mut rx) = client("ws://127.0.0.1:1", json!({}));
|
||||
assert!(!c.tts("r1", "x", false));
|
||||
let e = next_matching(&mut rx, |_| true).await;
|
||||
assert_eq!(e, WorkerEvent::TtsDone(TtsDone { id: "r1".into(), chunks: 0, cancelled: false, failed: true }));
|
||||
}
|
||||
120
app/tests/worker_process.rs
Normal file
120
app/tests/worker_process.rs
Normal file
@@ -0,0 +1,120 @@
|
||||
//! The worker process with a stand-in `voice_worker` package run by a real Python, so the spawn line
|
||||
//! is the one the app uses. JARVIS_TEST_PYTHON overrides the Python (default python3 / python).
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::time::Duration;
|
||||
|
||||
use jarvis_app::worker_process::{ProcessEvent, WorkerProcess, WorkerProcessOptions};
|
||||
use serde_json::Value;
|
||||
|
||||
const FAKE_MAIN: &str = r#"
|
||||
import json, os, sys, time
|
||||
out = os.environ["FAKE_OUT"]
|
||||
runs = len(os.listdir(out))
|
||||
with open(os.path.join(out, f"run{runs}.json"), "w") as f:
|
||||
json.dump({"argv": sys.argv[1:], "token": os.environ.get("JARVIS_VOICE_TOKEN")}, f)
|
||||
if os.environ.get("FAKE_CRASH_FIRST") == "1" and runs == 0:
|
||||
sys.exit(3)
|
||||
print("loading models...", flush=True)
|
||||
print("READY 127.0.0.1:" + sys.argv[sys.argv.index("--port") + 1], flush=True)
|
||||
time.sleep(60)
|
||||
"#;
|
||||
|
||||
fn python() -> String {
|
||||
std::env::var("JARVIS_TEST_PYTHON").unwrap_or_else(|_| if cfg!(windows) { "python".into() } else { "python3".into() })
|
||||
}
|
||||
|
||||
struct Fake {
|
||||
root: PathBuf,
|
||||
out: PathBuf,
|
||||
}
|
||||
|
||||
impl Fake {
|
||||
fn new(tag: &str) -> Self {
|
||||
let root = std::env::temp_dir().join(format!("jarvis-fake-worker-{tag}-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&root);
|
||||
std::fs::create_dir_all(root.join("voice_worker")).unwrap();
|
||||
std::fs::write(root.join("voice_worker/__init__.py"), "").unwrap();
|
||||
std::fs::write(root.join("voice_worker/__main__.py"), FAKE_MAIN).unwrap();
|
||||
let out = root.join("out");
|
||||
std::fs::create_dir_all(&out).unwrap();
|
||||
Self { root, out }
|
||||
}
|
||||
fn run(&self, n: usize) -> Value {
|
||||
serde_json::from_str(&std::fs::read_to_string(self.out.join(format!("run{n}.json"))).unwrap()).unwrap()
|
||||
}
|
||||
fn opts(&self, crash_first: bool) -> WorkerProcessOptions {
|
||||
let mut o = WorkerProcessOptions::new(python(), &self.root, vec!["--melo-python".into(), "/x/python".into()]);
|
||||
o.env.insert("FAKE_OUT".into(), self.out.display().to_string());
|
||||
if crash_first {
|
||||
o.env.insert("FAKE_CRASH_FIRST".into(), "1".into());
|
||||
}
|
||||
o.restart_delay = Duration::from_millis(50);
|
||||
o
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Fake {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.root);
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for(rx: &mut tokio::sync::broadcast::Receiver<ProcessEvent>, f: impl Fn(&ProcessEvent) -> bool) -> ProcessEvent {
|
||||
tokio::time::timeout(Duration::from_secs(10), async {
|
||||
loop {
|
||||
let e = rx.recv().await.unwrap();
|
||||
if f(&e) {
|
||||
return e;
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("timed out")
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn runs_the_worker_on_loopback_with_the_token_only_in_the_environment() {
|
||||
let fake = Fake::new("env");
|
||||
let p = WorkerProcess::new(fake.opts(false)).unwrap();
|
||||
let mut rx = p.subscribe();
|
||||
p.start();
|
||||
wait_for(&mut rx, |e| *e == ProcessEvent::Ready).await;
|
||||
let run = fake.run(0);
|
||||
let port = p.port().to_string();
|
||||
let argv: Vec<String> = serde_json::from_value(run["argv"].clone()).unwrap();
|
||||
assert_eq!(argv, ["--host", "127.0.0.1", "--port", &port, "--melo-python", "/x/python"]);
|
||||
assert_eq!(run["token"], p.token.as_str());
|
||||
assert!(p.token.len() >= 40);
|
||||
assert!(!argv.join(" ").contains(&p.token));
|
||||
assert!(p.url().starts_with("ws://127.0.0.1:"));
|
||||
p.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_dead_worker_restarts_on_the_same_port_with_the_same_token() {
|
||||
let fake = Fake::new("restart");
|
||||
let p = WorkerProcess::new(fake.opts(true)).unwrap();
|
||||
let mut rx = p.subscribe();
|
||||
p.start();
|
||||
assert_eq!(wait_for(&mut rx, |e| matches!(e, ProcessEvent::Exited { .. })).await, ProcessEvent::Exited { code: Some(3) });
|
||||
wait_for(&mut rx, |e| *e == ProcessEvent::Ready).await;
|
||||
assert_eq!(p.restarts(), 1);
|
||||
assert_eq!(fake.run(1)["argv"], fake.run(0)["argv"]);
|
||||
assert_eq!(fake.run(1)["token"], fake.run(0)["token"]);
|
||||
p.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stop_ends_the_worker_for_good() {
|
||||
let fake = Fake::new("stop");
|
||||
let p = WorkerProcess::new(fake.opts(false)).unwrap();
|
||||
let mut rx = p.subscribe();
|
||||
p.start();
|
||||
wait_for(&mut rx, |e| *e == ProcessEvent::Ready).await;
|
||||
p.stop().await;
|
||||
assert!(matches!(rx.recv().await.unwrap(), ProcessEvent::Exited { .. }));
|
||||
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||
assert!(rx.try_recv().is_err(), "no restart after stop");
|
||||
assert_eq!(p.restarts(), 0);
|
||||
}
|
||||
110
app/tests/worker_protocol.rs
Normal file
110
app/tests/worker_protocol.rs
Normal file
@@ -0,0 +1,110 @@
|
||||
//! The worker client and playback against the real Python voice worker server with fake engines, so
|
||||
//! both sides of the protocol are checked together. Needs a Python with the worker's requirements:
|
||||
//! JARVIS_VOICE_PYTHON=<venv python> cargo test --test worker_protocol
|
||||
|
||||
use std::path::Path;
|
||||
use std::time::Duration;
|
||||
|
||||
use jarvis_app::playback::{ClipSink, Playback, PlaybackEvent};
|
||||
use jarvis_app::worker_client::{WorkerClient, WorkerClientOptions, WorkerEvent};
|
||||
use jarvis_app::worker_process::free_port;
|
||||
use serde_json::json;
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
|
||||
const TOKEN: &str = "protocol-test-token";
|
||||
|
||||
/// 20 ms of a 48 kHz stereo tone whose amplitude and pitch (x1000 Hz) are `level`.
|
||||
fn frame(level: f64) -> Vec<i16> {
|
||||
(0..960)
|
||||
.flat_map(|i| {
|
||||
let v = (level * (2.0 * std::f64::consts::PI * level * 1000.0 * i as f64 / 48000.0).sin() * 32767.0) as i16;
|
||||
[v, v]
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Sink(Vec<usize>);
|
||||
impl ClipSink for Sink {
|
||||
fn play(&mut self, pcm: Vec<u8>) {
|
||||
self.0.push(pcm.len());
|
||||
}
|
||||
fn stop(&mut self) {}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn audio_and_tts_round_trip_through_the_python_server() {
|
||||
let Ok(python) = std::env::var("JARVIS_VOICE_PYTHON") else {
|
||||
eprintln!("skipped: set JARVIS_VOICE_PYTHON");
|
||||
return;
|
||||
};
|
||||
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
|
||||
let port = free_port().unwrap();
|
||||
let mut child = tokio::process::Command::new(python)
|
||||
.arg(root.join("tests/fixtures/fake_engine_worker.py"))
|
||||
.arg(port.to_string())
|
||||
.env("PYTHONPATH", root.join("../src"))
|
||||
.env("JARVIS_VOICE_TOKEN", TOKEN)
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.kill_on_drop(true)
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let mut lines = BufReader::new(child.stdout.take().unwrap()).lines();
|
||||
while let Some(l) = lines.next_line().await.unwrap() {
|
||||
if l.starts_with("READY") {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let mut o = WorkerClientOptions::new(format!("ws://127.0.0.1:{port}"), TOKEN);
|
||||
o.stt = json!({"partials": false, "hotwords": ["자비스"]}).as_object().unwrap().clone();
|
||||
let (c, mut rx) = WorkerClient::new(o);
|
||||
c.start();
|
||||
let ok = c.wait_ready(Duration::from_secs(5)).await.expect("ready");
|
||||
assert!(ok.stt_ready && ok.tts_ready);
|
||||
assert_eq!(ok.stt["hotwords"], json!(["자비스"]));
|
||||
|
||||
// Two Discord-format speakers; the second leaves mid-utterance and still gets its final.
|
||||
let (a, b) = ("111111111111111111", "222222222222222222");
|
||||
for _ in 0..30 {
|
||||
c.send_audio(a, &frame(0.9));
|
||||
c.send_audio(b, &frame(0.6));
|
||||
}
|
||||
c.speaking_end(a);
|
||||
c.speaker_leave(b);
|
||||
let mut finals = std::collections::HashMap::new();
|
||||
tokio::time::timeout(Duration::from_secs(5), async {
|
||||
while finals.len() < 2 {
|
||||
if let Some(WorkerEvent::Final(f)) = rx.recv().await {
|
||||
finals.insert(f.speaker.clone(), f);
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("finals");
|
||||
assert_eq!((finals[a].text.as_str(), finals[a].end_reason.as_str(), finals[a].utt), ("level 0.9", "speaking_end", 1));
|
||||
assert_eq!((finals[b].text.as_str(), finals[b].end_reason.as_str()), ("level 0.6", "speaking_end"));
|
||||
|
||||
// TTS sentences play in order through Playback; 10 samples per character, stereo, 2 bytes each.
|
||||
let mut p = Playback::new(c.clone(), Sink::default());
|
||||
p.say("첫 번째 문장입니다. 두 번째 문장은 조금 더 깁니다.", false);
|
||||
let end = tokio::time::timeout(Duration::from_secs(5), async {
|
||||
loop {
|
||||
match rx.recv().await.unwrap() {
|
||||
WorkerEvent::TtsAudio { id, seq, pcm } => {
|
||||
p.audio(&id, seq, pcm);
|
||||
p.clip_ended();
|
||||
}
|
||||
WorkerEvent::TtsDone(d) => p.done(&d),
|
||||
_ => {}
|
||||
}
|
||||
if let Some(PlaybackEvent::End(e)) = p.take_events().into_iter().find(|e| matches!(e, PlaybackEvent::End(_))) {
|
||||
return e;
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("speech end");
|
||||
assert_eq!((end.played, end.cancelled, end.failed), (2, false, false));
|
||||
c.close();
|
||||
let _ = child.kill().await;
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2023",
|
||||
"lib": ["ES2023"],
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"sourceMap": true,
|
||||
"types": ["node"]
|
||||
},
|
||||
"include": ["src"]
|
||||
}
|
||||
@@ -1,10 +0,0 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ["test/**/*.test.ts"],
|
||||
pool: "forks",
|
||||
poolOptions: { forks: { maxForks: 2, minForks: 1 } },
|
||||
testTimeout: 15_000,
|
||||
},
|
||||
});
|
||||
@@ -50,14 +50,14 @@ v1도 STT는 이미 faster-whisper입니다. 느린 진짜 이유는 엔진이
|
||||
```
|
||||
[게임PC (Windows 11) 한 대에서 전부 실행]
|
||||
|
||||
1. Jarvis 관리 앱 (exe, Electron + TypeScript)
|
||||
1. Jarvis 관리 앱 (exe, Rust. 화면은 Tauri 2 + Windows 기본 WebView2)
|
||||
├─ GUI: 대시보드 / 봇 / 채널·호출 / 채팅 채널 / 대화 기억 / 모델 / 기능
|
||||
│ / 게임 모드 / PC 제어 / 화면 보기 / 데이터 / 로그
|
||||
├─ 서비스 관리자: 아래 프로세스를 띄우고, 감시하고, 죽으면 재시작
|
||||
├─ 디스코드 음성 코어 (discord.js + @discordjs/voice + DAVE 복호화)
|
||||
├─ 디스코드 음성 코어 (serenity + songbird + DAVE 복호화, Opus 해독은 앱이 직접)
|
||||
│ 화자별 Opus 수신 → PCM → 음성 워커로 전달 / TTS 음성 재생
|
||||
├─ 채팅 채널 연결 (등록한 채널 읽기·보내기·수정·검색)
|
||||
├─ 브라우저 에이전트 (Playwright + 봇용 크롬창, 백그라운드 가능)
|
||||
├─ 브라우저 에이전트 (크롬 개발자 프로토콜(CDP) + 봇용 크롬창, 백그라운드 가능)
|
||||
├─ 게임·안티치트 감지 (게임 모드 전환, 자동 입력 차단)
|
||||
└─ PC 제어 모듈 (전체·모니터별·창별 화면 캡처, agent-desktop으로 프로그램 조작,
|
||||
접근성 정보가 없는 화면만 키보드/마우스 입력)
|
||||
@@ -80,10 +80,10 @@ v1도 STT는 이미 faster-whisper입니다. 느린 진짜 이유는 엔진이
|
||||
|
||||
| 부분 | 선택 | 이유 |
|
||||
|---|---|---|
|
||||
| GUI 앱 | Electron + TypeScript (React UI) | 디스코드 음성 코어(discord.js), Playwright, 화면 캡처가 모두 Node 생태계라 한 언어로 묶입니다. v1 봇 코드(TypeScript)를 옮길 수 있습니다. Tauri는 가볍지만 Rust와 Node 사이드카를 따로 둬야 해서 복잡해집니다. |
|
||||
| 관리 앱 | Rust (화면은 Tauri 2) | 처음에는 Electron + TypeScript로 정했지만 더 빠르고 가벼운 진짜 앱으로 바꿈(사용자 결정, 2026-11-11). 디스코드 음성(serenity, songbird, DAVE), 브라우저 조종(CDP), 화면 캡처(Windows Graphics Capture)를 모두 Rust로 할 수 있어 Node 사이드카가 필요 없습니다. 화면은 Tauri 2가 Windows에 이미 있는 WebView2로 그리므로 크로미움을 따로 싣지 않습니다. 같은 음성 코어를 이 서버에서 띄웠을 때 실행 파일 하나, 메모리 약 16MB였습니다(Electron은 프로세스 5개). songbird 0.6.1은 DAVE로 받은 소리를 해독할 때 조각 3분의 1가량을 깨뜨려서, 암호 풀기만 맡기고 순서 맞추기와 Opus 해독은 앱이 직접 합니다 ([discord_core.spec.md](../../app/src/discord_core.spec.md)). |
|
||||
| 음성 워커 | Python (faster-whisper, MeloTTS, 기존 jarvis) | STT·TTS 모델과 기존 두뇌 코드(메모리, 도구, 스펙 문서)가 Python입니다. 다시 짜는 것보다 재사용이 안전합니다. |
|
||||
| LLM 엔진 | Strata (MIT) | Windows 설치 지원, RTX 50 지원, `--data-dir`로 저장 위치 지정, 헤드리스 실행(`--no-tui`). 앱에 포함해 배포해도 라이선스 문제가 없습니다. |
|
||||
| PC 프로그램 조작 | agent-desktop (Apache-2.0) | Windows UI Automation으로 버튼·입력칸을 읽고 직접 누릅니다. 좌표 클릭보다 정확하고, 커서와 포커스를 뺏지 않습니다. 실행 파일 하나라 설치 파일에 넣기 쉽습니다. ([pc-features.md](pc-features.md#키보드마우스-제어와-안티치트)) |
|
||||
| LLM 예비 엔진 | OpenAI 호환 주소 | Strata에 문제가 생기면 같은 모델(GGUF)을 llama.cpp 같은 다른 로컬 엔진으로 돌려 연결합니다. 모델은 바꾸지 않습니다. |
|
||||
| Python 런타임 | uv + 독립 실행형 Python 3.12 | 사용자가 Python을 따로 깔 필요가 없고 시스템 Python과 섞이지 않습니다. 전부 데이터 폴더 안에 설치됩니다. |
|
||||
| 설치 파일 | NSIS (electron-builder) | 설치 파일은 약 200MB 내외로 두고, 모델과 런타임은 첫 실행 때 받습니다. |
|
||||
| 설치 파일 | NSIS (Tauri 번들러) | 설치 파일은 약 200MB 내외로 두고, 모델과 런타임은 첫 실행 때 받습니다. |
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
| 백그라운드 수집 대상 | 기본은 비워 둠. 요청이 있을 때 추가 |
|
||||
| Go Live 송출(셀프봇) | 뺌. 대신 봇용 크롬창 화면공유 선택 표시 설정 ([pc-features.md](pc-features.md#봇용-크롬창과-디스코드-화면공유)) |
|
||||
| 저장소 처리 | v1 마지막 상태(`bb60c4d`)를 원격 저장소 `legacy` 브랜치로 보존하고 같은 저장소에서 v2 시작. v1 README는 [docs/v1/README.md](../v1/README.md)로 옮김 |
|
||||
| 관리 앱 언어 | Rust. 화면(M5)은 Tauri 2로 Windows 기본 WebView2에 그림. 처음 정한 Electron + TypeScript보다 빠르고 가벼운 진짜 앱으로 만들기 위해 바꿈(사용자 결정, 2026-11-11). M2 음성 코어는 Rust로 다시 씀 ([architecture.md](architecture.md#기술-선택과-이유)) |
|
||||
| 호출 방식 기본값 | 호출어로 부를 때만 응답 |
|
||||
| 지정한 사람 설정 | "지정한 사람만 응답"과 "지정한 사람은 호출어 없이 대답"을 각각 켜기/끄기. 기본은 둘 다 꺼짐 ([conversation.md](conversation.md#호출어와-호출-방식)) |
|
||||
| 동시 대화 응답 정책 기본값 | 처음 대화만 |
|
||||
|
||||
@@ -309,6 +309,6 @@ LLM과 STT를 함께 켠 측정(M0 `gamemode_stt`, 게임 없음). STT를 먼저
|
||||
|
||||
## 백그라운드 크로미움으로 정보 수집
|
||||
|
||||
- 봇용 크롬(데이터 폴더 안)을 Playwright로 조종합니다. 웹 검색, 페이지 읽기, 예약 수집에 씁니다.
|
||||
- 봇용 크롬(데이터 폴더 안)을 크롬 개발자 프로토콜(CDP)로 조종합니다. 웹 검색, 페이지 읽기, 예약 수집에 씁니다.
|
||||
- 기존 webSearch 도구의 보안 장치를 그대로 가져갑니다. 내부망 접근을 막고, 웹 내용은 명령이 아닌 데이터로 다룹니다.
|
||||
- 예약 수집은 기능 틀(사이트·검색어·RSS를 주기적으로 모으는 기능)만 만들고 **수집 대상은 비워 둡니다**. 수집 결과는 데이터 폴더에만 저장합니다.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|---|---|---|
|
||||
| M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. 가상 모니터 확인을 뺀 모든 항목 측정 완료(2026-11-10). 가상 모니터 확인은 M6에서 합니다. 2026-11-10에 추가한 `desktop` 단계(agent-desktop)는 측정을 마쳤고, 자동 성능 조절용 `gamemode` 단계는 세 번 쟀고, `gamemode_stt`(LLM과 STT를 함께 켬, STT GPU·CPU)와 `gamemode_live`(실제 게임을 켬) 단계도 측정을 마쳤습니다(2026-11-10). 결과로 게임 중 최소 배치는 전문가 2GB로 정했고, 평소 배치에 STT 자리를 남기는 것과 게임 중 LLM이 답하는 동안 부분 전사 간격을 늘리는 것(게임이 없을 때는 그대로)도 정했습니다(사용자 결정, 2026-11-10). `gamemode_tune` 단계도 측정을 마쳤고(2026-11-11), 게임 중 LLM이 답하는 동안에는 부분 전사를 멈추기로 정했습니다. GPU 95% 제한은 측정에서 사용률을 낮추지 못해 기본값을 끔으로 바꿨고(사용자 결정, 2026-11-11), `video_drops` 단계(Jarvis를 돌리는 동안 유튜브 영상의 끊긴 프레임, 직접 실행)도 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 영상이 끊기지 않았고 제한이 끊김을 줄이지 못해 설정을 뺐습니다. 대신 게임 화면의 프레임을 직접 재는 `game_frames` 단계(게임을 켠 채로 Jarvis를 끈 때·쉴 때·바쁠 때와 우선순위를 낮춘 양보 상태, 직접 실행, 관리자 권한)를 넣었고 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 게임의 평균 fps와 1% low가 5% 넘게 떨어지지 않아 양보는 넣지 않기로 했습니다. 남은 M0 측정은 가상 모니터 확인(M6)입니다. 측정 키트: [scripts/m0/](../../scripts/m0/README.md) |
|
||||
| M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과. 구현과 테스트 완료(2026-11-11): `src/voice_worker/`, 스펙은 [voice_worker.spec.md](../../src/voice_worker/voice_worker.spec.md). 이 서버의 RTX 5050에서 10명이 겹쳐 말하는 실시간 입력(48kHz 스테레오, M0 한국어 샘플 8개를 10명에게 나눠 줌, 절반은 디스코드 speaking 종료, 절반은 무음으로 끝남)을 넣어 10명 모두 최종 전사를 받았고 문자 오류율은 평균 4%, 최대 19%("일곱 시"를 "7시"로 적은 것), 말 끝 판정부터 최종 전사 전송까지 중앙값 0.64초, 최대 1.56초였습니다. TTS는 첫 문장 소리가 0.2초에 나왔고, 그 소리를 다시 넣으면 같은 문장으로 받아 적혔습니다. 샘플이 MeloTTS 합성 음성이라 실제 사람 녹음과 게임PC 수치는 아직 없습니다 |
|
||||
| M2 | 디스코드 음성 코어 이전 | v1 봇 코드를 Electron 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기). 구현과 테스트 완료(2026-11-11): `app/`(Electron 45 + TypeScript), 스펙은 [discord_core.spec.md](../../app/src/core/discord_core.spec.md). 앱이 음성 워커를 띄우고(실행마다 새 토큰, 죽으면 다시 시작) 연결이 끊기면 다시 붙습니다. 음성 채널에 들어갈 때마다 전사 고지를 채널 채팅에 올립니다. 실제 디스코드에서 봇 두 개로 확인(Electron 런타임, 이 서버의 RTX 5050): 다른 봇이 채널에 튼 녹음 문장 5개를 모두 받아 적었고(문자 오류율 4개 0%, "일곱 시"를 "7시"로 적은 1개 12.5%), 말이 끝난 뒤 최종 전사까지 0.66~0.83초였습니다. 자비스가 말한 문장을 상대 봇이 받아 다시 전사하니 오류율 3.4%였고, 정지 명령 뒤 0.02초 만에 소리가 끊겼습니다. 채팅 채널은 새 글 받기, 최근 글 읽기, 보내기(2,000자 넘으면 나눔, 아무도 호출하지 않음)가 됩니다. 사람이 직접 말한 소리와 게임PC(Windows)에서는 아직 확인하지 않았고, 음성으로 "자비스, 멈춰"는 호출 판정이 필요해 M4에서 합니다 |
|
||||
| M2 | 디스코드 음성 코어 이전 | v1 봇 기능을 새 앱 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기). 구현과 테스트 완료(2026-11-11): `app/`(Rust, serenity + songbird), 스펙은 [discord_core.spec.md](../../app/src/discord_core.spec.md). 처음 Electron + TypeScript로 만든 것을 사용자 요청으로 Rust로 다시 씀. 앱이 음성 워커를 띄우고(실행마다 새 토큰, 죽으면 다시 시작) 연결이 끊기면 다시 붙습니다. 음성 채널에 들어갈 때마다 전사 고지를 채널 채팅에 올립니다. songbird 0.6.1의 DAVE 수신 해독이 조각 3분의 1가량을 깨뜨려서(20ms마다 음이 바뀌는 시험 신호로 확인, 보내기는 정상) 받은 패킷의 순서 맞추기와 Opus 해독은 앱이 직접 합니다. 실제 디스코드에서 봇 두 개로 확인(이 서버의 RTX 5050): 다른 봇이 채널에 튼 녹음 문장 5개를 모두 받아 적었고(문자 오류율 3개 0%, "기록해"를 "비록해"로 적은 1개 5%, "일곱 시"를 "7시"로 적은 1개 12.5%), 말이 끝난 뒤 최종 전사까지 0.71~0.90초였습니다. 자비스가 말한 문장을 상대 봇이 받아 다시 전사하니 오류율 3.4%였고, 정지 명령 뒤 0.08초 만에 소리가 끊겼습니다. 채팅 채널은 새 글 받기, 최근 글 읽기, 보내기(2,000자 넘으면 나눔, 아무도 호출하지 않음)가 됩니다. 사람이 직접 말한 소리와 게임PC(Windows) 빌드는 아직 확인하지 않았고, 음성으로 "자비스, 멈춰"는 호출 판정이 필요해 M4에서 합니다 |
|
||||
| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과 |
|
||||
| M4 | 왕복 완성과 지연 튜닝 | 호출어·호출 방식·지정한 사람 설정, 동시 대화 응답 정책 4가지(기본 "처음 대화만")와 정책별 끼어들기 규칙. 말 끝부터 첫 소리까지 실측 1.5초 이내, 10명 채널 부하 테스트 |
|
||||
| M5 | 관리 앱 기본 | 첫 실행 마법사, 데이터 폴더, 런타임·모델 자동 설치, 대시보드, 봇/채널·호출/채팅 채널(디스코드 채널 이름·주제 반영 포함)/대화 기억/모델 설정, 로그 화면(보기·복사·다운로드) |
|
||||
|
||||
Reference in New Issue
Block a user