feat(app): M2 Discord voice core in the Electron app
- app/ (Electron 45 + TypeScript): the core joins one voice channel, decodes each speaker's Opus to 48 kHz stereo and streams it to the voice worker, sends speaking_end and speaker_leave, and emits speech_start/partial/final with the speaker's display name. - Sentence playback from the worker's TTS, first come first served, with stop that silences the clip at once and cancels unfinished requests; slash command join/leave/stop (create-or-update by name). - Transcription notice posted on every join; other bots ignored by default; at most maxSpeakers people transcribed at once. - Registered chat channels: live message events, history reads, sends split at 2,000 characters with mentions disabled; Message Content intent checked before login. - The app starts the Python voice worker with a fresh token in the environment, restarts it with backoff, and the worker client reconnects and fails in-flight TTS on a lost connection. - Spec in app/src/core/discord_core.spec.md; Vitest unit tests, a protocol test against the real Python worker server with fake engines, and an opt-in live test with two bots on real Discord. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -29,6 +29,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
|
||||
| `src/jarvis/memory/summariser.spec.md` | Diary summariser prompt contract, hygiene rules (deflection, attribution, topic separation), post-process scrub, and bulk-sweep clean button | Two-layer defence: prompt + deterministic scrub; corrupted summaries poison every downstream consumer |
|
||||
| `src/jarvis/memory/recall_gate.spec.md` | Deterministic skip-enrichment heuristic when the hot window covers a follow-up | Fail-open; language-agnostic via `\w{3,}` + `re.UNICODE`; planner intent always wins |
|
||||
| `src/voice_worker/voice_worker.spec.md` | v2 voice worker: multi-speaker streaming STT (shared Whisper, batched) and sentence TTS over a local WebSocket | Loopback only with an app token; no transcript text in logs; finals before partials; language-agnostic |
|
||||
| `app/src/core/discord_core.spec.md` | v2 Discord voice core in the Electron app: join/leave, per-speaker receive to the voice worker, sentence playback and stop, slash command, registered chat channels (read/send), voice worker process | Hears only the channel it is in and reads only registered channels; join notice on every join; chat is data and sent text never pings; no transcript or chat text in logs |
|
||||
|
||||
The LLM contexts graph at `docs/llm_contexts.md` maps every LLM call in the app (model, gating, inputs, outputs, limits, flow). Keep it up-to-date at all times: any change that adds, removes, or alters an LLM context (model resolution, timeout, cap, prompt source, gating flag, data-flow edge) must update `docs/llm_contexts.md` in the same PR.
|
||||
|
||||
|
||||
2
app/.gitignore
vendored
Normal file
2
app/.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
node_modules/
|
||||
dist/
|
||||
2424
app/package-lock.json
generated
Normal file
2424
app/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
29
app/package.json
Normal file
29
app/package.json
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "jarvis-app",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Jarvis v2 Windows app: Discord voice core, voice worker supervisor (Electron + TypeScript)",
|
||||
"type": "module",
|
||||
"main": "dist/main.js",
|
||||
"scripts": {
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"typecheck": "tsc -p tsconfig.json --noEmit && tsc -p test/tsconfig.json",
|
||||
"test": "vitest run --maxWorkers=2",
|
||||
"start": "npm run build && electron .",
|
||||
"start:node": "npm run build && node dist/cli.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"@discordjs/voice": "^0.19.2",
|
||||
"discord.js": "^14.28.1",
|
||||
"opusscript": "^0.0.8",
|
||||
"prism-media": "^1.3.5",
|
||||
"ws": "^8.22.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.19.0",
|
||||
"@types/ws": "^8.18.1",
|
||||
"electron": "^45.3.0",
|
||||
"typescript": "~5.9.3",
|
||||
"vitest": "^3.2.4"
|
||||
}
|
||||
}
|
||||
23
app/src/cli.ts
Normal file
23
app/src/cli.ts
Normal file
@@ -0,0 +1,23 @@
|
||||
/** Run the app's services without Electron (development and tests): node dist/cli.js <data folder> */
|
||||
import { errorLog } from "./core/log.js";
|
||||
import { startServices } from "./services.js";
|
||||
|
||||
const dataDir = process.argv[2] ?? process.env.JARVIS_DATA_DIR;
|
||||
if (!dataDir) {
|
||||
errorLog("usage: node dist/cli.js <data folder> (or set JARVIS_DATA_DIR)");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
startServices(dataDir).then((services) => {
|
||||
let stopping = false;
|
||||
const stop = () => {
|
||||
if (stopping) return;
|
||||
stopping = true;
|
||||
void services.stop().finally(() => process.exit(0));
|
||||
};
|
||||
process.on("SIGINT", stop);
|
||||
process.on("SIGTERM", stop);
|
||||
}, (e: Error) => {
|
||||
errorLog(`could not start: ${e.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
92
app/src/core/chat.ts
Normal file
92
app/src/core/chat.ts
Normal file
@@ -0,0 +1,92 @@
|
||||
/** Registered chat channels: which ones the bot may read or write, and message splitting. */
|
||||
import type { ChatChannelSettings } from "./settings.js";
|
||||
|
||||
/** Discord's message length limit. */
|
||||
export const MESSAGE_LIMIT = 2000;
|
||||
|
||||
export interface ChatMessage {
|
||||
channelId: string;
|
||||
messageId: string;
|
||||
authorId: string;
|
||||
authorName: string;
|
||||
authorIsBot: boolean;
|
||||
/** Written by this bot. Live events never carry these; history reads do. */
|
||||
fromSelf: boolean;
|
||||
/** What the person wrote. Data, never instructions (privacy.md). */
|
||||
content: string;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export class ChatAccessError extends Error {}
|
||||
|
||||
export class ChatChannels {
|
||||
private byId: Map<string, ChatChannelSettings>;
|
||||
|
||||
constructor(channels: ChatChannelSettings[]) {
|
||||
this.byId = new Map(channels.map((c) => [c.id, c]));
|
||||
}
|
||||
|
||||
get(id: string): ChatChannelSettings | undefined {
|
||||
return this.byId.get(id);
|
||||
}
|
||||
|
||||
list(): ChatChannelSettings[] {
|
||||
return [...this.byId.values()];
|
||||
}
|
||||
|
||||
/** Some channel is readable, so the bot needs the message intents. */
|
||||
get anyReadable(): boolean {
|
||||
return this.list().some((c) => c.read);
|
||||
}
|
||||
|
||||
canRead(id: string): boolean {
|
||||
return this.byId.get(id)?.read === true;
|
||||
}
|
||||
|
||||
canSend(id: string): boolean {
|
||||
return this.byId.get(id)?.send === true;
|
||||
}
|
||||
|
||||
assertRead(id: string): void {
|
||||
if (!this.byId.has(id)) throw new ChatAccessError(`channel ${id} is not a registered chat channel`);
|
||||
if (!this.canRead(id)) throw new ChatAccessError(`reading channel ${id} is turned off`);
|
||||
}
|
||||
|
||||
assertSend(id: string): void {
|
||||
if (!this.byId.has(id)) throw new ChatAccessError(`channel ${id} is not a registered chat channel`);
|
||||
if (!this.canSend(id)) throw new ChatAccessError(`sending to channel ${id} is turned off`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Split text into messages of at most `limit` characters (UTF-16 units, as Discord counts),
|
||||
* breaking at the last newline, else the last space, else hard. Nothing is lost except the
|
||||
* newline or space a break happens at.
|
||||
*/
|
||||
export function splitMessage(text: string, limit = MESSAGE_LIMIT): string[] {
|
||||
const out: string[] = [];
|
||||
let rest = text;
|
||||
while (rest.length > limit) {
|
||||
const window = rest.slice(0, limit + 1);
|
||||
let cut = window.lastIndexOf("\n");
|
||||
if (cut <= 0) cut = window.lastIndexOf(" ");
|
||||
if (cut <= 0) {
|
||||
cut = limit;
|
||||
// Do not split a surrogate pair.
|
||||
const code = rest.charCodeAt(cut - 1);
|
||||
if (code >= 0xd800 && code <= 0xdbff) cut--;
|
||||
out.push(rest.slice(0, cut));
|
||||
rest = rest.slice(cut);
|
||||
} else {
|
||||
out.push(rest.slice(0, cut));
|
||||
rest = rest.slice(cut + 1);
|
||||
}
|
||||
}
|
||||
if (rest.length) out.push(rest);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Intent bits in the application flags (GATEWAY_MESSAGE_CONTENT and its _LIMITED form). */
|
||||
export function hasMessageContentIntent(applicationFlags: number): boolean {
|
||||
return (applicationFlags & ((1 << 18) | (1 << 19))) !== 0;
|
||||
}
|
||||
502
app/src/core/discordCore.ts
Normal file
502
app/src/core/discordCore.ts
Normal file
@@ -0,0 +1,502 @@
|
||||
/**
|
||||
* The Discord voice core (app/src/core/discord_core.spec.md).
|
||||
*
|
||||
* One bot, one server, one voice channel at a time. Hears each speaker separately and streams their
|
||||
* PCM to the voice worker, speaks text through the worker's TTS, and reads and writes the registered
|
||||
* chat channels. It has no assistant logic: it reports what was heard and does what it is told.
|
||||
*/
|
||||
import { EventEmitter } from "node:events";
|
||||
import { Readable } from "node:stream";
|
||||
import {
|
||||
ApplicationIntegrationType,
|
||||
Client,
|
||||
Events,
|
||||
GatewayIntentBits,
|
||||
InteractionContextType,
|
||||
MessageFlags,
|
||||
REST,
|
||||
Routes,
|
||||
SlashCommandBuilder,
|
||||
type ChatInputCommandInteraction,
|
||||
type GuildMember,
|
||||
type Message,
|
||||
type VoiceBasedChannel,
|
||||
type VoiceState,
|
||||
} from "discord.js";
|
||||
import {
|
||||
AudioPlayerStatus,
|
||||
createAudioPlayer,
|
||||
createAudioResource,
|
||||
EndBehaviorType,
|
||||
entersState,
|
||||
joinVoiceChannel,
|
||||
NoSubscriberBehavior,
|
||||
StreamType,
|
||||
VoiceConnectionStatus,
|
||||
type AudioPlayer,
|
||||
type AudioResource,
|
||||
type VoiceConnection,
|
||||
} from "@discordjs/voice";
|
||||
import OpusScript from "opusscript";
|
||||
import { ChatAccessError, ChatChannels, hasMessageContentIntent, splitMessage, type ChatMessage } from "./chat.js";
|
||||
import { debugLog, errorLog } from "./log.js";
|
||||
import { Playback, type PlaybackEnd } from "./playback.js";
|
||||
import type { Settings } from "./settings.js";
|
||||
import { SpeakerGate } from "./speakers.js";
|
||||
import type { Final, VoiceWorkerClient } from "./workerClient.js";
|
||||
|
||||
/** Discord voice is 48 kHz stereo PCM after Opus decoding; the worker is told the same in hello. */
|
||||
export const DISCORD_PCM = { rate: 48000, channels: 2 } as const;
|
||||
|
||||
export interface HeardSpeech {
|
||||
speakerId: string;
|
||||
speakerName: string;
|
||||
utt: number;
|
||||
}
|
||||
|
||||
export interface HeardFinal extends HeardSpeech {
|
||||
text: string;
|
||||
endReason: string;
|
||||
audioMs: number;
|
||||
waitMs: number;
|
||||
sttMs: number;
|
||||
}
|
||||
|
||||
export interface CoreEvents {
|
||||
ready: [{ userId: string; tag: string }];
|
||||
joined: [{ guildId: string; channelId: string }];
|
||||
left: [{ guildId: string; channelId: string; reason: string }];
|
||||
speech_start: [HeardSpeech];
|
||||
partial: [HeardSpeech & { text: string }];
|
||||
final: [HeardFinal];
|
||||
speak_start: [string];
|
||||
speak_end: [PlaybackEnd];
|
||||
chat: [ChatMessage];
|
||||
}
|
||||
|
||||
interface SpeakerPipe {
|
||||
stream: Readable;
|
||||
decoder: OpusScript;
|
||||
}
|
||||
|
||||
/** A live voice connection: per-speaker receive and the playback queue. */
|
||||
class VoiceSession {
|
||||
readonly connection: VoiceConnection;
|
||||
readonly player: AudioPlayer;
|
||||
readonly playback: Playback;
|
||||
private current: AudioResource | null = null;
|
||||
private pipes = new Map<string, SpeakerPipe>();
|
||||
private destroyed = false;
|
||||
badPackets = 0;
|
||||
|
||||
constructor(
|
||||
readonly guildId: string,
|
||||
public channelId: string,
|
||||
channel: VoiceBasedChannel,
|
||||
private readonly worker: VoiceWorkerClient,
|
||||
private readonly gate: SpeakerGate,
|
||||
private readonly shouldHear: (userId: string) => boolean,
|
||||
) {
|
||||
this.connection = joinVoiceChannel({
|
||||
channelId: channel.id,
|
||||
guildId: channel.guild.id,
|
||||
adapterCreator: channel.guild.voiceAdapterCreator,
|
||||
selfDeaf: false,
|
||||
selfMute: false,
|
||||
});
|
||||
this.player = createAudioPlayer({ behaviors: { noSubscriber: NoSubscriberBehavior.Pause } });
|
||||
this.connection.subscribe(this.player);
|
||||
this.playback = new Playback(worker, {
|
||||
play: (pcm) => {
|
||||
const resource = createAudioResource(Readable.from([pcm]), { inputType: StreamType.Raw });
|
||||
this.current = resource;
|
||||
this.player.play(resource);
|
||||
},
|
||||
stop: () => {
|
||||
this.current = null;
|
||||
this.player.stop(true);
|
||||
},
|
||||
});
|
||||
this.player.on("stateChange", (oldState, newState) => {
|
||||
// Only the end of the clip we started counts; a stop() already moved on.
|
||||
if (oldState.status !== AudioPlayerStatus.Idle && newState.status === AudioPlayerStatus.Idle
|
||||
&& "resource" in oldState && oldState.resource === this.current) {
|
||||
this.current = null;
|
||||
this.playback.clipEnded();
|
||||
}
|
||||
});
|
||||
this.player.on("error", (e) => errorLog(`playback error: ${e.message}`, "voice"));
|
||||
const speaking = this.connection.receiver.speaking;
|
||||
speaking.on("start", (userId) => this.onSpeakingStart(userId));
|
||||
speaking.on("end", (userId) => this.onSpeakingEnd(userId));
|
||||
}
|
||||
|
||||
async ready(timeoutMs = 20_000): Promise<void> {
|
||||
await entersState(this.connection, VoiceConnectionStatus.Ready, timeoutMs);
|
||||
}
|
||||
|
||||
private onSpeakingStart(userId: string): void {
|
||||
if (this.destroyed || !this.shouldHear(userId)) return;
|
||||
if (!this.gate.start(userId)) {
|
||||
debugLog(`speaker ${userId} not heard: ${this.gate.count} already speaking`, "voice");
|
||||
return;
|
||||
}
|
||||
this.subscribe(userId);
|
||||
}
|
||||
|
||||
private onSpeakingEnd(userId: string): void {
|
||||
if (this.gate.end(userId)) this.worker.speakingEnd(userId);
|
||||
}
|
||||
|
||||
/** One long-lived Opus subscription per speaker; PCM goes to the worker only while they are heard. */
|
||||
private subscribe(userId: string): void {
|
||||
if (this.pipes.has(userId)) return;
|
||||
const stream = this.connection.receiver.subscribe(userId, { end: { behavior: EndBehaviorType.Manual } });
|
||||
const decoder = new OpusScript(DISCORD_PCM.rate, DISCORD_PCM.channels, OpusScript.Application.VOIP);
|
||||
const pipe: SpeakerPipe = { stream, decoder };
|
||||
this.pipes.set(userId, pipe);
|
||||
stream.on("data", (packet: Buffer) => {
|
||||
if (!this.gate.isActive(userId)) return;
|
||||
let pcm: Buffer;
|
||||
try {
|
||||
pcm = decoder.decode(packet);
|
||||
} catch {
|
||||
this.badPackets++;
|
||||
return;
|
||||
}
|
||||
this.worker.sendAudio(userId, pcm);
|
||||
});
|
||||
stream.on("error", (e) => debugLog(`receive stream for ${userId} failed: ${e.message}`, "voice"));
|
||||
stream.once("close", () => {
|
||||
if (this.pipes.get(userId) !== pipe) return;
|
||||
this.pipes.delete(userId);
|
||||
decoder.delete();
|
||||
// A packet that failed to decrypt destroys the stream; pick the speaker up again at once.
|
||||
if (!this.destroyed && this.gate.isActive(userId)) this.subscribe(userId);
|
||||
});
|
||||
}
|
||||
|
||||
/** The person left the channel: finish their utterance and free their place in the worker. */
|
||||
speakerLeft(userId: string): void {
|
||||
const wasHeard = this.gate.end(userId);
|
||||
const pipe = this.pipes.get(userId);
|
||||
if (pipe) {
|
||||
this.pipes.delete(userId);
|
||||
pipe.stream.destroy();
|
||||
pipe.decoder.delete();
|
||||
}
|
||||
if (wasHeard || pipe) this.worker.speakerLeave(userId);
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
if (this.destroyed) return;
|
||||
this.destroyed = true;
|
||||
this.playback.stop();
|
||||
for (const [userId, pipe] of this.pipes) {
|
||||
pipe.stream.destroy();
|
||||
pipe.decoder.delete();
|
||||
this.worker.speakerLeave(userId);
|
||||
}
|
||||
this.pipes.clear();
|
||||
this.gate.clear();
|
||||
this.player.stop(true);
|
||||
if (this.connection.state.status !== VoiceConnectionStatus.Destroyed) this.connection.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
export class DiscordCore extends EventEmitter<CoreEvents> {
|
||||
private _client: Client | null = null;
|
||||
private session: VoiceSession | null = null;
|
||||
private gate: SpeakerGate;
|
||||
private chat: ChatChannels;
|
||||
private names = new Map<string, string>();
|
||||
/** The application has the Message Content intent; without it chat channels cannot be read. */
|
||||
chatContent = false;
|
||||
|
||||
constructor(private readonly settings: Settings, private readonly token: string,
|
||||
private readonly worker: VoiceWorkerClient) {
|
||||
super();
|
||||
this.gate = new SpeakerGate(settings.voice.maxSpeakers);
|
||||
this.chat = new ChatChannels(settings.chatChannels);
|
||||
worker.on("speech_start", (m) => this.emit("speech_start", this.heard(m.speaker, m.utt)));
|
||||
worker.on("partial", (m) => this.emit("partial", { ...this.heard(m.speaker, m.utt), text: m.text }));
|
||||
worker.on("final", (m) => this.onFinal(m));
|
||||
worker.on("tts_audio", (f) => this.session?.playback.audio(f.id, f.seq, f.pcm));
|
||||
worker.on("tts_done", (d) => this.session?.playback.done(d));
|
||||
}
|
||||
|
||||
/** The discord.js client; exists once start() has begun (intents are chosen at login). */
|
||||
get client(): Client {
|
||||
if (!this._client) throw new Error("the Discord core has not been started");
|
||||
return this._client;
|
||||
}
|
||||
|
||||
/** Log in, register the slash command, and join the auto-join channel if one is set. */
|
||||
async start(): Promise<void> {
|
||||
const rest = new REST({ version: "10" }).setToken(this.token);
|
||||
const app = (await rest.get(Routes.currentApplication())) as { id: string; flags?: number };
|
||||
this.chatContent = hasMessageContentIntent(app.flags ?? 0);
|
||||
const intents = [GatewayIntentBits.Guilds, GatewayIntentBits.GuildVoiceStates];
|
||||
if (this.chat.anyReadable) {
|
||||
intents.push(GatewayIntentBits.GuildMessages);
|
||||
if (this.chatContent) intents.push(GatewayIntentBits.MessageContent);
|
||||
else errorLog("Message Content Intent is off for this bot; chat channels cannot be read. "
|
||||
+ "Turn it on in the Discord Developer Portal (Bot > Privileged Gateway Intents).", "discord");
|
||||
}
|
||||
const client = new Client({ intents });
|
||||
this._client = client;
|
||||
client.on(Events.InteractionCreate, (i) => {
|
||||
if (i.isChatInputCommand()) void this.onCommand(i);
|
||||
});
|
||||
client.on(Events.VoiceStateUpdate, (o, n) => this.onVoiceState(o, n));
|
||||
client.on(Events.MessageCreate, (m) => this.onMessage(m));
|
||||
client.on(Events.Error, (e) => errorLog(`gateway error: ${e.message}`, "discord"));
|
||||
const ready = new Promise<void>((resolve) => client.once(Events.ClientReady, () => resolve()));
|
||||
await client.login(this.token);
|
||||
await ready;
|
||||
debugLog(`logged in as ${client.user!.id} chatContent=${this.chatContent}`, "discord");
|
||||
this.emit("ready", { userId: client.user!.id, tag: client.user!.tag });
|
||||
try {
|
||||
await this.registerCommand(rest, app.id);
|
||||
} catch (e) {
|
||||
// Voice and chat still work; the command can be registered on the next start.
|
||||
errorLog(`could not register the slash command: ${(e as Error).message}`, "discord");
|
||||
}
|
||||
const auto = this.settings.discord.voiceChannelId;
|
||||
if (auto) {
|
||||
try {
|
||||
await this.join(auto);
|
||||
} catch (e) {
|
||||
errorLog(`auto-join ${auto} failed: ${(e as Error).message}`, "voice");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async registerCommand(rest: REST, appId: string): Promise<void> {
|
||||
const guildId = this.settings.discord.guildId;
|
||||
if (!guildId) {
|
||||
errorLog("discord.guildId is not set; the slash command is not registered", "discord");
|
||||
return;
|
||||
}
|
||||
const cmd = new SlashCommandBuilder()
|
||||
.setName(this.settings.discord.commandName)
|
||||
.setDescription("자비스 음성 비서")
|
||||
.setContexts(InteractionContextType.Guild)
|
||||
.setIntegrationTypes(ApplicationIntegrationType.GuildInstall)
|
||||
.addSubcommand((s) => s.setName("join").setDescription("내가 있는 음성 채널로 자비스를 부릅니다"))
|
||||
.addSubcommand((s) => s.setName("leave").setDescription("자비스를 음성 채널에서 내보냅니다"))
|
||||
.addSubcommand((s) => s.setName("stop").setDescription("자비스가 하던 말을 멈춥니다"));
|
||||
// POST creates or updates this one command by name and leaves the app's other commands alone.
|
||||
await rest.post(Routes.applicationGuildCommands(appId, guildId), { body: cmd.toJSON() });
|
||||
debugLog(`slash command /${cmd.name} registered in ${guildId}`, "discord");
|
||||
}
|
||||
|
||||
// --- voice ------------------------------------------------------------------------------------
|
||||
|
||||
get voiceChannelId(): string | null {
|
||||
return this.session?.channelId ?? null;
|
||||
}
|
||||
|
||||
/** Join a voice channel of the configured server (leaving the current one) and post the notice. */
|
||||
async join(channelId: string): Promise<void> {
|
||||
const channel = await this.client.channels.fetch(channelId);
|
||||
if (!channel?.isVoiceBased()) throw new Error(`channel ${channelId} is not a voice channel`);
|
||||
const guildId = this.settings.discord.guildId;
|
||||
if (guildId && channel.guild.id !== guildId) throw new Error(`channel ${channelId} is not in the configured server`);
|
||||
if (this.session?.channelId === channel.id) return;
|
||||
this.leave("moved");
|
||||
const session = new VoiceSession(channel.guild.id, channel.id, channel, this.worker, this.gate,
|
||||
(id) => this.shouldHear(channel.guild.id, id));
|
||||
this.session = session;
|
||||
session.playback.on("start", (id) => this.emit("speak_start", id));
|
||||
session.playback.on("end", (e) => this.emit("speak_end", e));
|
||||
session.connection.on(VoiceConnectionStatus.Disconnected, async () => {
|
||||
try {
|
||||
// Moved to another channel or a brief network drop: the library reconnects on its own.
|
||||
await Promise.race([
|
||||
entersState(session.connection, VoiceConnectionStatus.Signalling, 5000),
|
||||
entersState(session.connection, VoiceConnectionStatus.Connecting, 5000),
|
||||
]);
|
||||
} catch {
|
||||
if (this.session === session) this.leave("disconnected");
|
||||
}
|
||||
});
|
||||
try {
|
||||
await session.ready();
|
||||
} catch (e) {
|
||||
if (this.session === session) this.leave("connect_failed");
|
||||
throw new Error(`could not connect to voice channel ${channelId}: ${(e as Error).message}`);
|
||||
}
|
||||
debugLog(`joined voice ${channel.id}`, "voice");
|
||||
this.emit("joined", { guildId: channel.guild.id, channelId: channel.id });
|
||||
try {
|
||||
await channel.send({ content: this.settings.voice.joinNotice, allowedMentions: { parse: [] } });
|
||||
} catch (e) {
|
||||
errorLog(`could not post the transcription notice in ${channel.id}: ${(e as Error).message}`, "voice");
|
||||
}
|
||||
}
|
||||
|
||||
/** Leave the voice channel. False if not in one. */
|
||||
leave(reason = "command"): boolean {
|
||||
const s = this.session;
|
||||
if (!s) return false;
|
||||
this.session = null;
|
||||
s.destroy();
|
||||
debugLog(`left voice ${s.channelId} reason=${reason} bad_packets=${s.badPackets}`, "voice");
|
||||
this.emit("left", { guildId: s.guildId, channelId: s.channelId, reason });
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Speak text in the voice channel. Returns the request id, or null when not in a voice channel. */
|
||||
say(text: string, more = false): string | null {
|
||||
return this.session ? this.session.playback.say(text, more) : null;
|
||||
}
|
||||
|
||||
/** More text for a request started with say(..., true). */
|
||||
continueText(id: string, text: string, more = false): void {
|
||||
this.session?.playback.continueText(id, text, more);
|
||||
}
|
||||
|
||||
/** Stop speaking at once and drop everything queued. Returns how many requests were stopped. */
|
||||
stop(): number {
|
||||
return this.session?.playback.stop() ?? 0;
|
||||
}
|
||||
|
||||
get speaking(): boolean {
|
||||
return this.session?.playback.busy ?? false;
|
||||
}
|
||||
|
||||
private shouldHear(guildId: string, userId: string): boolean {
|
||||
if (userId === this.client.user?.id) return false;
|
||||
if (this.settings.voice.listenToBots) return true;
|
||||
const user = this.client.users.cache.get(userId) ?? this.client.guilds.cache.get(guildId)?.members.cache.get(userId)?.user;
|
||||
// Unknown users are people: the voice state that put them in the channel would have cached a bot.
|
||||
return !user?.bot;
|
||||
}
|
||||
|
||||
private onVoiceState(oldState: VoiceState, newState: VoiceState): void {
|
||||
const s = this.session;
|
||||
if (!s || oldState.guild.id !== s.guildId) return;
|
||||
const userId = newState.id;
|
||||
if (userId === this.client.user?.id) {
|
||||
// Moved by someone else: follow the new channel id.
|
||||
if (newState.channelId && newState.channelId !== s.channelId) s.channelId = newState.channelId;
|
||||
return;
|
||||
}
|
||||
if (newState.member) this.names.set(userId, displayName(newState.member));
|
||||
if (oldState.channelId === s.channelId && newState.channelId !== s.channelId) s.speakerLeft(userId);
|
||||
}
|
||||
|
||||
private heard(speakerId: string, utt: number): HeardSpeech {
|
||||
return { speakerId, speakerName: this.nameOf(speakerId), utt };
|
||||
}
|
||||
|
||||
private nameOf(userId: string): string {
|
||||
const cached = this.names.get(userId);
|
||||
if (cached) return cached;
|
||||
const guildId = this.session?.guildId ?? this.settings.discord.guildId;
|
||||
const member = guildId ? this.client.guilds.cache.get(guildId)?.members.cache.get(userId) : undefined;
|
||||
const name = member ? displayName(member) : this.client.users.cache.get(userId)?.displayName ?? userId;
|
||||
this.names.set(userId, name);
|
||||
return name;
|
||||
}
|
||||
|
||||
private onFinal(m: Final): void {
|
||||
debugLog(`final speaker=${m.speaker} utt=${m.utt} reason=${m.end_reason} chars=${m.text.length} `
|
||||
+ `audio_ms=${m.audio_ms} wait_ms=${m.wait_ms}`, "voice");
|
||||
this.emit("final", {
|
||||
...this.heard(m.speaker, m.utt), text: m.text, endReason: m.end_reason, audioMs: m.audio_ms,
|
||||
waitMs: m.wait_ms, sttMs: m.stt_ms,
|
||||
});
|
||||
}
|
||||
|
||||
// --- slash command ----------------------------------------------------------------------------
|
||||
|
||||
private async onCommand(i: ChatInputCommandInteraction): Promise<void> {
|
||||
if (i.commandName !== this.settings.discord.commandName) return;
|
||||
const guildId = this.settings.discord.guildId;
|
||||
if (!i.inGuild() || (guildId && i.guildId !== guildId)) return;
|
||||
const flags = this.settings.discord.ephemeralReplies ? MessageFlags.Ephemeral : undefined;
|
||||
const sub = i.options.getSubcommand();
|
||||
debugLog(`command ${sub} from ${i.user.id}`, "discord");
|
||||
try {
|
||||
if (sub === "join") {
|
||||
const member = i.member as GuildMember | null;
|
||||
const channelId = member?.voice?.channelId
|
||||
?? i.guild?.voiceStates.cache.get(i.user.id)?.channelId ?? null;
|
||||
if (!channelId) {
|
||||
await i.reply({ content: "먼저 음성 채널에 들어간 뒤 다시 불러 주세요.", flags });
|
||||
return;
|
||||
}
|
||||
await i.deferReply({ flags });
|
||||
await this.join(channelId);
|
||||
await i.editReply("음성 채널에 들어왔습니다.");
|
||||
} else if (sub === "leave") {
|
||||
await i.reply({ content: this.leave() ? "음성 채널에서 나갔습니다." : "음성 채널에 있지 않습니다.", flags });
|
||||
} else if (sub === "stop") {
|
||||
await i.reply({ content: this.stop() ? "멈췄습니다." : "지금은 말하고 있지 않습니다.", flags });
|
||||
}
|
||||
} catch (e) {
|
||||
errorLog(`command ${sub} failed: ${(e as Error).message}`, "discord");
|
||||
const content = "처리하지 못했습니다. 잠시 뒤 다시 시도해 주세요.";
|
||||
if (i.deferred || i.replied) await i.editReply(content).catch(() => {});
|
||||
else await i.reply({ content, flags }).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
// --- chat channels ----------------------------------------------------------------------------
|
||||
|
||||
private toChat(m: Message): ChatMessage {
|
||||
return {
|
||||
channelId: m.channelId,
|
||||
messageId: m.id,
|
||||
authorId: m.author.id,
|
||||
authorName: m.member ? displayName(m.member) : m.author.displayName,
|
||||
authorIsBot: m.author.bot,
|
||||
fromSelf: m.author.id === this.client.user?.id,
|
||||
content: m.content,
|
||||
createdAt: m.createdAt.toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
private onMessage(m: Message): void {
|
||||
if (!this.chatContent || !this.chat.canRead(m.channelId) || m.author.id === this.client.user?.id) return;
|
||||
debugLog(`chat message in ${m.channelId} from ${m.author.id} chars=${m.content.length}`, "chat");
|
||||
this.emit("chat", this.toChat(m));
|
||||
}
|
||||
|
||||
/** The latest `limit` messages of a registered channel, oldest first. */
|
||||
async readChat(channelId: string, limit = 20): Promise<ChatMessage[]> {
|
||||
this.chat.assertRead(channelId);
|
||||
if (!this.chatContent) throw new ChatAccessError("the bot's Message Content Intent is off");
|
||||
const channel = await this.client.channels.fetch(channelId);
|
||||
if (!channel?.isTextBased()) throw new ChatAccessError(`channel ${channelId} is not a text channel`);
|
||||
const messages = await channel.messages.fetch({ limit: Math.max(1, Math.min(100, limit)) });
|
||||
return [...messages.values()].sort((a, b) => a.createdTimestamp - b.createdTimestamp).map((m) => this.toChat(m));
|
||||
}
|
||||
|
||||
/** Send text to a registered channel, split at Discord's length limit. Returns the message ids. */
|
||||
async sendChat(channelId: string, text: string): Promise<string[]> {
|
||||
this.chat.assertSend(channelId);
|
||||
if (!text.trim()) throw new ChatAccessError("nothing to send");
|
||||
const channel = await this.client.channels.fetch(channelId);
|
||||
if (!channel?.isSendable()) throw new ChatAccessError(`channel ${channelId} is not a channel the bot can send to`);
|
||||
const ids: string[] = [];
|
||||
for (const part of splitMessage(text)) {
|
||||
// Never ping anyone from text the assistant wrote.
|
||||
const sent = await channel.send({ content: part, allowedMentions: { parse: [] } });
|
||||
ids.push(sent.id);
|
||||
}
|
||||
debugLog(`chat sent to ${channelId} parts=${ids.length} chars=${text.length}`, "chat");
|
||||
return ids;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.leave("shutdown");
|
||||
await this._client?.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
function displayName(member: GuildMember): string {
|
||||
return member.displayName || member.user.globalName || member.user.username;
|
||||
}
|
||||
163
app/src/core/discord_core.spec.md
Normal file
163
app/src/core/discord_core.spec.md
Normal file
@@ -0,0 +1,163 @@
|
||||
# Discord voice core spec (v2 M2)
|
||||
|
||||
The Discord voice core is the part of the Windows app (Electron main process, TypeScript) that
|
||||
connects the bot to Discord: it joins a voice channel, hears each person separately, streams their
|
||||
audio to the voice worker for transcription, plays the worker's speech back into the channel, and reads
|
||||
and writes the registered chat channels. It has no assistant logic. It reports what it heard and does
|
||||
what it is told; the brain (M3) and call detection and response policies (M4) sit on top of it.
|
||||
Design background: `docs/v2/architecture.md`, `docs/v2/conversation.md`, `docs/v2/privacy.md`. The
|
||||
worker side of the link is `src/voice_worker/voice_worker.spec.md`.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Only where the bot is.** The bot hears only the voice channel it is in and reads only the chat
|
||||
channels registered in settings with reading on. Every join posts the transcription notice in the
|
||||
voice channel's chat.
|
||||
- **Chat is data.** Messages read from chat channels are handed on as data, never acted on as
|
||||
commands. Text the bot sends never pings anyone (`allowedMentions: { parse: [] }`).
|
||||
- **No text in logs.** Logs carry ids, sizes, counts and timings, never what anyone said or wrote, or
|
||||
what the bot says. Debug logs (`debugLog`, on with `JARVIS_DEBUG=1`) go at joins and leaves, command
|
||||
use, finals, speech requests and stops, chat events and sends, and worker connection changes.
|
||||
- **Language-agnostic.** Nothing in the core matches words. Wake words are only passed to the STT as a
|
||||
hint until call detection (M4) uses them.
|
||||
- **Testable without Discord.** The pieces with rules (playback queue, speaker limit, chat access and
|
||||
splitting, settings, worker link, worker process) are plain classes tested with fakes; the Discord
|
||||
glue is tested on real Discord by an opt-in live test.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
app/
|
||||
├── package.json # Electron + TypeScript; npm test, npm run build, npm start
|
||||
├── src/main.ts # Electron main: starts the services, stops them on quit (no window until M5)
|
||||
├── src/cli.ts # The same services under plain Node: node dist/cli.js <data folder>
|
||||
├── src/services.ts # Settings -> voice worker process -> worker client -> Discord core
|
||||
└── src/core/
|
||||
├── discordCore.ts # Discord client, voice session, slash command, chat channels
|
||||
├── playback.ts # Speech queue (TTS requests -> sentence clips -> player)
|
||||
├── speakers.ts # Who is transcribed right now (max speakers)
|
||||
├── chat.ts # Chat channel access rules and message splitting
|
||||
├── workerClient.ts # WebSocket client for the voice worker
|
||||
├── workerProcess.ts # Starts and restarts the Python voice worker
|
||||
├── frames.ts # Binary frame layout shared with voice_worker/protocol.py
|
||||
├── settings.ts # settings.json loading and validation
|
||||
└── log.ts # debugLog / errorLog
|
||||
```
|
||||
|
||||
## Starting
|
||||
|
||||
`startServices(dataDir)` reads `<dataDir>/config/settings.json`, starts the voice worker process,
|
||||
connects to it once it prints `READY`, and starts the Discord core. Until the setup wizard (M5) stores
|
||||
it encrypted, the bot token comes from `JARVIS_DISCORD_TOKEN` and the data folder from
|
||||
`JARVIS_DATA_DIR` (Electron) or the first argument (`cli.js`). The Electron entry holds a single
|
||||
instance lock and stops every service before it quits.
|
||||
|
||||
## Settings
|
||||
|
||||
`<data folder>/config/settings.json`. Every key is optional; missing keys take the defaults. A bad
|
||||
value stops the app with a message naming the key. Unknown keys are ignored.
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `discord.guildId` | `null` | The one server. The slash command is registered there; commands and channels from other servers are ignored. |
|
||||
| `discord.voiceChannelId` | `null` | Join this voice channel on start ("자동 입장"). |
|
||||
| `discord.commandName` | `"자비스"` | Slash command name (Discord rule: 1-32 lowercase letters, digits, `-`, `_`). |
|
||||
| `discord.ephemeralReplies` | `true` | Command replies only the caller sees. |
|
||||
| `voice.maxSpeakers` | `10` | Most people transcribed at once, 1-10. |
|
||||
| `voice.listenToBots` | `false` | Also transcribe other bots (music bots are ignored by default). |
|
||||
| `voice.joinNotice` | "자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다." | Posted in the voice channel's chat on every join. Must not be empty. |
|
||||
| `voice.wakeWords` | `["자비스"]` | Sent to the worker as STT `hotwords`. |
|
||||
| `voice.stt` | `{}` | Other worker STT settings (`partials`, `end_silence_ms`, `final_beam`, `language`, ...); the worker validates them. |
|
||||
| `chatChannels` | `[]` | `{id, name, description, read (true), send (true)}` per channel. |
|
||||
| `voiceWorker.python` | `python3` (`python.exe` on Windows) | Python of the worker venv. |
|
||||
| `voiceWorker.cwd` | the repo's `src/` | Folder holding the `voice_worker` package. |
|
||||
| `voiceWorker.args` | `[]` | Extra worker arguments, e.g. `["--melo-python", "<MeloTTS venv python>"]`. |
|
||||
|
||||
## Voice worker process
|
||||
|
||||
The app starts `python -m voice_worker --host 127.0.0.1 --port <port> <args>` in `voiceWorker.cwd`.
|
||||
Each launch makes a new random token (32 bytes) and passes it only in `JARVIS_VOICE_TOKEN`. The port is
|
||||
a free loopback port chosen once at start. If the worker exits, it is started again on the same port
|
||||
with the same token after 1 s, doubling to at most 30 s while it keeps failing (reset once it prints
|
||||
`READY`). Stopping sends SIGTERM and kills it after 5 s.
|
||||
|
||||
## Worker link
|
||||
|
||||
One WebSocket to the worker, `hello` with the token, the Discord PCM format both ways
|
||||
(48 kHz stereo), and the STT settings (`voice.stt` with `hotwords` = `voice.wakeWords`). If the
|
||||
connection drops, the client reconnects with backoff (0.5 s doubling to 5 s) and says hello again with
|
||||
the same settings, including changes made since with `configure()`. While disconnected, audio frames
|
||||
are dropped (not queued), and every TTS request in flight ends as failed, so playback moves on. A TTS
|
||||
request the worker refuses (`tts_unavailable`) also ends as failed. TTS audio for request ids the client
|
||||
did not open is ignored.
|
||||
|
||||
## Voice
|
||||
|
||||
One voice session at a time, in the configured server. Joining another channel leaves the current one.
|
||||
|
||||
- **Join** (slash command `join`, auto-join, or `join(channelId)`): connect with DAVE end-to-end
|
||||
encryption (the library's default), not deafened, then post `voice.joinNotice` in the voice
|
||||
channel's chat. A notice that cannot be posted (no permission) is logged and the bot stays.
|
||||
- **Hearing**: Discord's speaking start for a user starts one long-lived Opus subscription for them,
|
||||
decoded with opusscript to 48 kHz stereo PCM and sent to the worker in 20 ms frames under the user's
|
||||
ID. The bot's own audio is never heard; other bots are skipped unless `voice.listenToBots`.
|
||||
- At most `voice.maxSpeakers` people are transcribed at once. Someone who starts while the limit is
|
||||
reached is not heard until their next speaking start after a place frees up.
|
||||
- Discord's speaking end (no packets for 100 ms) sends `speaking_end`.
|
||||
- A subscription that fails (a packet that cannot be decrypted closes it) is opened again at once
|
||||
while the person is still being heard.
|
||||
- When a person leaves the channel, the bot sends `speaker_leave`, so their utterance finishes and
|
||||
the worker frees their place.
|
||||
- **Events** for the layers above: `speech_start`, `partial` and `final` with the speaker's ID and
|
||||
server display name; `speak_start` and `speak_end`; `joined` and `left` (with a reason: `command`,
|
||||
`moved`, `disconnected`, `connect_failed`, `shutdown`).
|
||||
- **Speaking**: `say(text, more?)` queues a TTS request and returns its id (`null` when not in a voice
|
||||
channel); `continueText(id, text, more?)` adds text to an open request. Each sentence the worker
|
||||
sends becomes one clip; clips play in sentence order as soon as they arrive, and requests play first
|
||||
come, first served. A sentence that failed is skipped and the rest still play. `speak_end` reports
|
||||
how many sentences played and whether the request was cancelled or failed.
|
||||
- **Stop** (slash command `stop`, or `stop()`): the current clip stops at once (no padding frames),
|
||||
every queued request is dropped, and unfinished requests are cancelled at the worker; each ends with
|
||||
`cancelled: true`. Stopping by voice ("자비스, 멈춰") needs call detection and comes in M4.
|
||||
- **Leave** (slash command `leave`, or `leave()`): stop speaking, close every subscription (sending
|
||||
`speaker_leave`), and disconnect. A connection lost for more than 5 s (kicked, channel deleted) also
|
||||
ends the session. When someone moves the bot to another channel, the session follows it.
|
||||
|
||||
## Slash command
|
||||
|
||||
One guild command, `/<commandName>` with subcommands `join` (the caller's voice channel), `leave`
|
||||
and `stop`, registered on every start with a create-or-update by name, so the app's other commands in
|
||||
the server are left alone. A failed registration (for example a bot invited without the
|
||||
`applications.commands` scope) is logged and the bot runs on without it. Usable only in the configured
|
||||
server. Replies are in Korean, ephemeral by
|
||||
default. Renaming `commandName` leaves the old command registered until it is removed (M5 setup).
|
||||
|
||||
## Chat channels
|
||||
|
||||
- The bot asks for the message intents only when some registered channel has reading on. Reading
|
||||
needs the Message Content intent, which is checked in the application's flags before login. When it
|
||||
is off, the bot still logs in, voice works, and an error explains how to turn it on; chat reads then
|
||||
fail with a clear error instead of returning empty messages.
|
||||
- **Live**: every new message in a readable registered channel, except the bot's own, is emitted as a
|
||||
`chat` event: channel, message and author IDs, author display name, whether the author is a bot, the
|
||||
text, and the time.
|
||||
- **History**: `readChat(channelId, limit)` returns the latest 1-100 messages, oldest first, with
|
||||
`fromSelf` marking the bot's own.
|
||||
- **Send**: `sendChat(channelId, text)` sends to a registered channel with sending on, split at
|
||||
Discord's 2,000-character limit at the last newline or space (never inside an emoji), and returns
|
||||
the message ids.
|
||||
- A channel that is not registered, or has the needed switch off, is refused with `ChatAccessError`.
|
||||
|
||||
## Tests
|
||||
|
||||
- `npm test` (Vitest): playback order, gaps, stop and cancel; speaker limit; chat access and
|
||||
splitting; settings; the worker link against a fake server (hello, events, reconnect, failed TTS);
|
||||
the worker process with a stand-in `voice_worker` package (token only in the environment, restart on
|
||||
the same port, stop).
|
||||
- `JARVIS_VOICE_PYTHON=<venv python> npm test` also runs the worker link against the real Python
|
||||
worker server with fake engines (`test/fixtures/fake_engine_worker.py`).
|
||||
- `test/live/discord.live.test.ts` (opt-in, `JARVIS_LIVE_DISCORD=1`, see the file header): two bots in
|
||||
one voice channel with the real worker on the GPU. It refuses to run if a person is in the channel
|
||||
or either bot is already in a voice channel. It checks command registration, transcription of
|
||||
recorded sentences played into the channel, speech the other bot hears and the STT reads back, stop,
|
||||
chat events, history, sending without pings, and leaving.
|
||||
32
app/src/core/frames.ts
Normal file
32
app/src/core/frames.ts
Normal file
@@ -0,0 +1,32 @@
|
||||
/** Binary frame layout shared with the Python voice worker (src/voice_worker/protocol.py). */
|
||||
|
||||
export const KIND_AUDIO = 0x01;
|
||||
export const KIND_TTS = 0x02;
|
||||
|
||||
function encodeId(value: string): Buffer {
|
||||
const raw = Buffer.from(value, "utf8");
|
||||
if (raw.length === 0 || raw.length > 255) throw new RangeError("id must be 1-255 bytes of UTF-8");
|
||||
return Buffer.concat([Buffer.from([raw.length]), raw]);
|
||||
}
|
||||
|
||||
/** 0x01 | u8 n | speaker id | PCM16LE in the hello "audio" format. */
|
||||
export function encodeAudioFrame(speaker: string, pcm: Buffer): Buffer {
|
||||
return Buffer.concat([Buffer.from([KIND_AUDIO]), encodeId(speaker), pcm]);
|
||||
}
|
||||
|
||||
export interface TtsFrame {
|
||||
id: string;
|
||||
seq: number;
|
||||
pcm: Buffer;
|
||||
}
|
||||
|
||||
/** 0x02 | u8 n | request id | u32 LE seq | PCM16LE in the hello "tts" format. Null if malformed. */
|
||||
export function decodeTtsFrame(frame: Buffer): TtsFrame | null {
|
||||
if (frame.length < 2 || frame[0] !== KIND_TTS) return null;
|
||||
const n = frame[1]!;
|
||||
const end = 2 + n;
|
||||
if (n === 0 || frame.length < end + 4) return null;
|
||||
const id = frame.subarray(2, end).toString("utf8");
|
||||
const seq = frame.readUInt32LE(end);
|
||||
return { id, seq, pcm: frame.subarray(end + 4) };
|
||||
}
|
||||
25
app/src/core/log.ts
Normal file
25
app/src/core/log.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
/**
|
||||
* Debug logging for the app, the TypeScript side of `src/jarvis/debug.py`.
|
||||
*
|
||||
* On only when JARVIS_DEBUG=1. Messages carry ids, sizes, counts and timings, never what anyone said
|
||||
* or wrote (no transcripts, no chat text, no reply text).
|
||||
*/
|
||||
export function debugLog(message: string, category = "app"): void {
|
||||
if (process.env.JARVIS_DEBUG !== "1") return;
|
||||
try {
|
||||
const pad = Math.max(0, 10 - category.length);
|
||||
const left = Math.floor(pad / 2);
|
||||
process.stderr.write(`[${" ".repeat(left)}${category}${" ".repeat(pad - left)}] ${message}\n`);
|
||||
} catch {
|
||||
/* logging must never break the app */
|
||||
}
|
||||
}
|
||||
|
||||
/** Errors are always printed; the same rule applies: no transcript or chat text in the message. */
|
||||
export function errorLog(message: string, category = "app"): void {
|
||||
try {
|
||||
process.stderr.write(`[${category}] ERROR ${message}\n`);
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
138
app/src/core/playback.ts
Normal file
138
app/src/core/playback.ts
Normal file
@@ -0,0 +1,138 @@
|
||||
/**
|
||||
* The bot's mouth: turns text into queued speech, one request after another.
|
||||
*
|
||||
* Each request goes to the voice worker as TTS; the worker sends back one PCM chunk per sentence.
|
||||
* Chunks play in order as soon as they arrive, so the first sentence plays while later ones are still
|
||||
* being made. Requests play first come, first served. stop() silences everything at once and cancels
|
||||
* what the worker has not made yet.
|
||||
*/
|
||||
import { EventEmitter } from "node:events";
|
||||
import { debugLog } from "./log.js";
|
||||
import type { TtsDone } from "./workerClient.js";
|
||||
|
||||
export interface TtsBackend {
|
||||
tts(id: string, text: string, more?: boolean): boolean;
|
||||
ttsCancel(id: string): void;
|
||||
}
|
||||
|
||||
/** Plays one PCM clip at a time; the owner calls Playback.clipEnded() when a clip finishes. */
|
||||
export interface ClipSink {
|
||||
play(pcm: Buffer): void;
|
||||
stop(): void;
|
||||
}
|
||||
|
||||
export interface PlaybackEnd {
|
||||
id: string;
|
||||
/** Sentences that were played. */
|
||||
played: number;
|
||||
cancelled: boolean;
|
||||
failed: boolean;
|
||||
}
|
||||
|
||||
interface Request {
|
||||
id: string;
|
||||
chunks: Map<number, Buffer>;
|
||||
next: number;
|
||||
done: boolean;
|
||||
failed: boolean;
|
||||
played: number;
|
||||
started: boolean;
|
||||
}
|
||||
|
||||
export class Playback extends EventEmitter<{ start: [string]; end: [PlaybackEnd] }> {
|
||||
private queue: Request[] = [];
|
||||
private byId = new Map<string, Request>();
|
||||
private playing = false;
|
||||
private counter = 0;
|
||||
|
||||
constructor(private readonly backend: TtsBackend, private readonly sink: ClipSink) {
|
||||
super();
|
||||
}
|
||||
|
||||
/** True while a clip is playing or a request is waiting for its audio. */
|
||||
get busy(): boolean {
|
||||
return this.playing || this.queue.length > 0;
|
||||
}
|
||||
|
||||
/** Queue `text` as a new request. With `more`, add text later with continueText(). Returns its id. */
|
||||
say(text: string, more = false): string {
|
||||
const id = `r${++this.counter}`;
|
||||
const req: Request = { id, chunks: new Map(), next: 0, done: false, failed: false, played: 0, started: false };
|
||||
this.queue.push(req);
|
||||
this.byId.set(id, req);
|
||||
debugLog(`say id=${id} chars=${text.length} more=${more} queued=${this.queue.length}`, "play");
|
||||
this.backend.tts(id, text, more);
|
||||
return id;
|
||||
}
|
||||
|
||||
/** More text for an open request (an LLM answer that is still streaming). */
|
||||
continueText(id: string, text: string, more = false): void {
|
||||
if (this.byId.has(id)) this.backend.tts(id, text, more);
|
||||
}
|
||||
|
||||
/** A sentence of audio arrived from the worker. */
|
||||
audio(id: string, seq: number, pcm: Buffer): void {
|
||||
const req = this.byId.get(id);
|
||||
if (!req) return;
|
||||
req.chunks.set(seq, pcm);
|
||||
this.pump();
|
||||
}
|
||||
|
||||
/** The worker finished a request (all sentences sent, cancelled, or failed). */
|
||||
done(d: TtsDone): void {
|
||||
const req = this.byId.get(d.id);
|
||||
if (!req) return;
|
||||
req.done = true;
|
||||
req.failed = d.failed;
|
||||
this.pump();
|
||||
}
|
||||
|
||||
/** The sink finished playing the current clip. */
|
||||
clipEnded(): void {
|
||||
this.playing = false;
|
||||
this.pump();
|
||||
}
|
||||
|
||||
/** Silence now: stop the clip, drop every queued request, cancel the worker's work. */
|
||||
stop(): number {
|
||||
const stopped = this.queue.splice(0);
|
||||
this.byId.clear();
|
||||
const wasPlaying = this.playing;
|
||||
this.playing = false;
|
||||
for (const req of stopped) if (!req.done) this.backend.ttsCancel(req.id);
|
||||
if (wasPlaying) this.sink.stop();
|
||||
for (const req of stopped) this.emit("end", { id: req.id, played: req.played, cancelled: true, failed: false });
|
||||
debugLog(`stop requests=${stopped.length} was_playing=${wasPlaying}`, "play");
|
||||
return stopped.length;
|
||||
}
|
||||
|
||||
private pump(): void {
|
||||
while (!this.playing) {
|
||||
const head = this.queue[0];
|
||||
if (!head) return;
|
||||
let pcm = head.chunks.get(head.next);
|
||||
if (!pcm && head.done && head.chunks.size) {
|
||||
// A sentence failed: skip its gap and play what did arrive.
|
||||
head.next = Math.min(...head.chunks.keys());
|
||||
pcm = head.chunks.get(head.next);
|
||||
}
|
||||
if (pcm) {
|
||||
head.chunks.delete(head.next);
|
||||
head.next++;
|
||||
head.played++;
|
||||
if (!head.started) {
|
||||
head.started = true;
|
||||
this.emit("start", head.id);
|
||||
}
|
||||
this.playing = true;
|
||||
this.sink.play(pcm);
|
||||
return;
|
||||
}
|
||||
if (!head.done) return;
|
||||
this.queue.shift();
|
||||
this.byId.delete(head.id);
|
||||
debugLog(`said id=${head.id} sentences=${head.played} failed=${head.failed}`, "play");
|
||||
this.emit("end", { id: head.id, played: head.played, cancelled: false, failed: head.failed });
|
||||
}
|
||||
}
|
||||
}
|
||||
179
app/src/core/settings.ts
Normal file
179
app/src/core/settings.ts
Normal file
@@ -0,0 +1,179 @@
|
||||
/**
|
||||
* App settings: `<data folder>/config/settings.json` (see app/src/core/discord_core.spec.md).
|
||||
*
|
||||
* Every key is optional in the file; missing keys take the defaults below. A bad value stops the app
|
||||
* with a message naming the key, rather than running with a setting the user did not choose.
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
export interface ChatChannelSettings {
|
||||
id: string;
|
||||
/** What the channel is for, as the user describes it; given to the assistant (M3). */
|
||||
name: string;
|
||||
description: string;
|
||||
/** The assistant may read this channel's messages. */
|
||||
read: boolean;
|
||||
/** The assistant may send messages to this channel. */
|
||||
send: boolean;
|
||||
}
|
||||
|
||||
export interface Settings {
|
||||
discord: {
|
||||
/** The one server the bot works in. Slash commands are registered there. */
|
||||
guildId: string | null;
|
||||
/** Voice channel to join on start ("자동 입장"); null waits for the join command. */
|
||||
voiceChannelId: string | null;
|
||||
commandName: string;
|
||||
/** Slash command replies only the caller sees. */
|
||||
ephemeralReplies: boolean;
|
||||
};
|
||||
voice: {
|
||||
/** Most people transcribed at once (1-10). Others are not heard until someone stops. */
|
||||
maxSpeakers: number;
|
||||
/** Transcribe other bots too (music bots are ignored by default). */
|
||||
listenToBots: boolean;
|
||||
/** Posted in the voice channel's chat on every join (privacy.md). */
|
||||
joinNotice: string;
|
||||
/** Wake words; passed to the STT as a hint now, used for call detection from M4. */
|
||||
wakeWords: string[];
|
||||
/** Voice worker STT settings (voice_worker.spec.md); the worker validates them. */
|
||||
stt: Record<string, unknown>;
|
||||
};
|
||||
chatChannels: ChatChannelSettings[];
|
||||
voiceWorker: {
|
||||
/** Python of the voice worker venv. */
|
||||
python: string;
|
||||
/** Directory holding the `voice_worker` package (the repo's `src/`). */
|
||||
cwd: string;
|
||||
/** Extra worker arguments, e.g. ["--melo-python", "<MeloTTS venv python>"]. */
|
||||
args: string[];
|
||||
};
|
||||
}
|
||||
|
||||
export class SettingsError extends Error {}
|
||||
|
||||
export const DEFAULT_JOIN_NOTICE =
|
||||
"자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다.";
|
||||
|
||||
const SNOWFLAKE = /^\d{17,20}$/;
|
||||
// Discord's rule for command names.
|
||||
const COMMAND_NAME = /^[-_\p{L}\p{N}\p{sc=Deva}\p{sc=Thai}]{1,32}$/u;
|
||||
|
||||
export function defaultSettings(): Settings {
|
||||
return {
|
||||
discord: { guildId: null, voiceChannelId: null, commandName: "자비스", ephemeralReplies: true },
|
||||
voice: { maxSpeakers: 10, listenToBots: false, joinNotice: DEFAULT_JOIN_NOTICE, wakeWords: ["자비스"], stt: {} },
|
||||
chatChannels: [],
|
||||
voiceWorker: {
|
||||
python: process.platform === "win32" ? "python.exe" : "python3",
|
||||
cwd: fileURLToPath(new URL("../../../src", import.meta.url)),
|
||||
args: [],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
type Obj = Record<string, unknown>;
|
||||
|
||||
function isObj(v: unknown): v is Obj {
|
||||
return typeof v === "object" && v !== null && !Array.isArray(v);
|
||||
}
|
||||
|
||||
function section(raw: Obj, key: string): Obj {
|
||||
const v = raw[key];
|
||||
if (v === undefined) return {};
|
||||
if (!isObj(v)) throw new SettingsError(`${key} must be an object`);
|
||||
return v;
|
||||
}
|
||||
|
||||
function pick<T>(obj: Obj, key: string, path: string, fallback: T, check: (v: unknown) => v is T, what: string): T {
|
||||
const v = obj[key];
|
||||
if (v === undefined) return fallback;
|
||||
if (!check(v)) throw new SettingsError(`${path}.${key} must be ${what}`);
|
||||
return v;
|
||||
}
|
||||
|
||||
const isStr = (v: unknown): v is string => typeof v === "string";
|
||||
const isBool = (v: unknown): v is boolean => typeof v === "boolean";
|
||||
const isSnowflakeOrNull = (v: unknown): v is string | null => v === null || (isStr(v) && SNOWFLAKE.test(v));
|
||||
const isStrList = (v: unknown): v is string[] => Array.isArray(v) && v.every(isStr);
|
||||
|
||||
/** Validate a parsed settings object over the defaults. */
|
||||
export function parseSettings(raw: unknown, defaults: Settings = defaultSettings()): Settings {
|
||||
if (!isObj(raw)) throw new SettingsError("settings must be a JSON object");
|
||||
const d = section(raw, "discord");
|
||||
const v = section(raw, "voice");
|
||||
const w = section(raw, "voiceWorker");
|
||||
|
||||
const commandName = pick(d, "commandName", "discord", defaults.discord.commandName, isStr, "a string");
|
||||
if (!COMMAND_NAME.test(commandName) || commandName !== commandName.toLowerCase()) {
|
||||
throw new SettingsError("discord.commandName must be 1-32 lowercase letters, digits, - or _");
|
||||
}
|
||||
const maxSpeakers = pick(v, "maxSpeakers", "voice", defaults.voice.maxSpeakers,
|
||||
(x): x is number => Number.isInteger(x) && (x as number) >= 1 && (x as number) <= 10, "an integer 1-10");
|
||||
const wakeWords = pick(v, "wakeWords", "voice", defaults.voice.wakeWords, isStrList, "a list of strings")
|
||||
.map((s) => s.trim()).filter(Boolean);
|
||||
const joinNotice = pick(v, "joinNotice", "voice", defaults.voice.joinNotice, isStr, "a string").trim();
|
||||
if (!joinNotice) throw new SettingsError("voice.joinNotice must not be empty: people must be told they are transcribed");
|
||||
if (joinNotice.length > 2000) throw new SettingsError("voice.joinNotice must be at most 2000 characters");
|
||||
|
||||
const chatRaw = raw.chatChannels ?? [];
|
||||
if (!Array.isArray(chatRaw)) throw new SettingsError("chatChannels must be a list");
|
||||
const seen = new Set<string>();
|
||||
const chatChannels = chatRaw.map((c, i): ChatChannelSettings => {
|
||||
const path = `chatChannels[${i}]`;
|
||||
if (!isObj(c)) throw new SettingsError(`${path} must be an object`);
|
||||
const id = c.id;
|
||||
if (!isStr(id) || !SNOWFLAKE.test(id)) throw new SettingsError(`${path}.id must be a Discord channel ID`);
|
||||
if (seen.has(id)) throw new SettingsError(`${path}.id is listed twice`);
|
||||
seen.add(id);
|
||||
return {
|
||||
id,
|
||||
name: pick(c, "name", path, "", isStr, "a string"),
|
||||
description: pick(c, "description", path, "", isStr, "a string"),
|
||||
read: pick(c, "read", path, true, isBool, "true or false"),
|
||||
send: pick(c, "send", path, true, isBool, "true or false"),
|
||||
};
|
||||
});
|
||||
|
||||
return {
|
||||
discord: {
|
||||
guildId: pick(d, "guildId", "discord", defaults.discord.guildId, isSnowflakeOrNull, "a Discord server ID or null"),
|
||||
voiceChannelId: pick(d, "voiceChannelId", "discord", defaults.discord.voiceChannelId, isSnowflakeOrNull,
|
||||
"a Discord channel ID or null"),
|
||||
commandName,
|
||||
ephemeralReplies: pick(d, "ephemeralReplies", "discord", defaults.discord.ephemeralReplies, isBool, "true or false"),
|
||||
},
|
||||
voice: {
|
||||
maxSpeakers,
|
||||
listenToBots: pick(v, "listenToBots", "voice", defaults.voice.listenToBots, isBool, "true or false"),
|
||||
joinNotice,
|
||||
wakeWords,
|
||||
stt: pick(v, "stt", "voice", defaults.voice.stt, isObj, "an object"),
|
||||
},
|
||||
chatChannels,
|
||||
voiceWorker: {
|
||||
python: pick(w, "python", "voiceWorker", defaults.voiceWorker.python, isStr, "a string"),
|
||||
cwd: pick(w, "cwd", "voiceWorker", defaults.voiceWorker.cwd, isStr, "a string"),
|
||||
args: pick(w, "args", "voiceWorker", defaults.voiceWorker.args, isStrList, "a list of strings"),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Read and validate the settings file. A missing file means all defaults. */
|
||||
export function loadSettings(path: string): Settings {
|
||||
let text: string;
|
||||
try {
|
||||
text = readFileSync(path, "utf8");
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException).code === "ENOENT") return defaultSettings();
|
||||
throw e;
|
||||
}
|
||||
let raw: unknown;
|
||||
try {
|
||||
raw = JSON.parse(text.replace(/^/, ""));
|
||||
} catch (e) {
|
||||
throw new SettingsError(`${path} is not valid JSON: ${(e as Error).message}`);
|
||||
}
|
||||
return parseSettings(raw);
|
||||
}
|
||||
35
app/src/core/speakers.ts
Normal file
35
app/src/core/speakers.ts
Normal file
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* Who is being transcribed right now. At most `max` people at once (settings voice.maxSpeakers);
|
||||
* someone who starts while the limit is reached is not heard until their next start after the
|
||||
* others free a place.
|
||||
*/
|
||||
export class SpeakerGate {
|
||||
private active = new Set<string>();
|
||||
|
||||
constructor(private max: number) {}
|
||||
|
||||
/** Discord says `id` started speaking. True if their audio goes to the STT. */
|
||||
start(id: string): boolean {
|
||||
if (this.active.has(id)) return true;
|
||||
if (this.active.size >= this.max) return false;
|
||||
this.active.add(id);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Discord says `id` stopped (or left). True if they were being transcribed. */
|
||||
end(id: string): boolean {
|
||||
return this.active.delete(id);
|
||||
}
|
||||
|
||||
isActive(id: string): boolean {
|
||||
return this.active.has(id);
|
||||
}
|
||||
|
||||
get count(): number {
|
||||
return this.active.size;
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
this.active.clear();
|
||||
}
|
||||
}
|
||||
278
app/src/core/workerClient.ts
Normal file
278
app/src/core/workerClient.ts
Normal file
@@ -0,0 +1,278 @@
|
||||
/**
|
||||
* WebSocket client for the Python voice worker (src/voice_worker/voice_worker.spec.md).
|
||||
*
|
||||
* Keeps one connection open, says hello with the app's token, and reconnects with backoff when the
|
||||
* worker restarts. While disconnected, audio is dropped (the worker would not know the speaker's
|
||||
* state anyway) and every open TTS request ends as failed so playback never waits forever.
|
||||
*/
|
||||
import { EventEmitter } from "node:events";
|
||||
import WebSocket from "ws";
|
||||
import { decodeTtsFrame, encodeAudioFrame, KIND_TTS } from "./frames.js";
|
||||
import { debugLog, errorLog } from "./log.js";
|
||||
|
||||
export interface PcmFormat {
|
||||
rate: number;
|
||||
channels: 1 | 2;
|
||||
}
|
||||
|
||||
export interface WorkerClientOptions {
|
||||
url: string;
|
||||
token: string;
|
||||
audio: PcmFormat;
|
||||
tts: PcmFormat;
|
||||
stt?: Record<string, unknown>;
|
||||
/** First reconnect delay; doubles up to maxBackoffMs. */
|
||||
backoffMs?: number;
|
||||
maxBackoffMs?: number;
|
||||
helloTimeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface HelloOk {
|
||||
stt_ready: boolean;
|
||||
tts_ready: boolean;
|
||||
stt: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface SpeechStart {
|
||||
speaker: string;
|
||||
utt: number;
|
||||
}
|
||||
export interface Partial {
|
||||
speaker: string;
|
||||
utt: number;
|
||||
text: string;
|
||||
}
|
||||
export interface Final {
|
||||
speaker: string;
|
||||
utt: number;
|
||||
text: string;
|
||||
end_reason: string;
|
||||
audio_ms: number;
|
||||
wait_ms: number;
|
||||
stt_ms: number;
|
||||
}
|
||||
export interface TtsDone {
|
||||
id: string;
|
||||
chunks: number;
|
||||
cancelled: boolean;
|
||||
failed: boolean;
|
||||
}
|
||||
export interface WorkerError {
|
||||
code: string;
|
||||
message: string;
|
||||
id?: string;
|
||||
}
|
||||
|
||||
export interface WorkerClientEvents {
|
||||
ready: [HelloOk];
|
||||
disconnected: [];
|
||||
speech_start: [SpeechStart];
|
||||
partial: [Partial];
|
||||
final: [Final];
|
||||
tts_audio: [{ id: string; seq: number; pcm: Buffer }];
|
||||
tts_done: [TtsDone];
|
||||
worker_error: [WorkerError];
|
||||
}
|
||||
|
||||
export class VoiceWorkerClient extends EventEmitter<WorkerClientEvents> {
|
||||
private ws: WebSocket | null = null;
|
||||
private ready = false;
|
||||
private closed = false;
|
||||
private retryTimer: NodeJS.Timeout | null = null;
|
||||
private backoff: number;
|
||||
private stt: Record<string, unknown>;
|
||||
/** TTS requests sent and not yet done, so a lost connection can end them. */
|
||||
private openTts = new Set<string>();
|
||||
private dropped = 0;
|
||||
private hello: HelloOk | null = null;
|
||||
|
||||
constructor(private readonly opts: WorkerClientOptions) {
|
||||
super();
|
||||
this.backoff = opts.backoffMs ?? 500;
|
||||
this.stt = { ...(opts.stt ?? {}) };
|
||||
}
|
||||
|
||||
get isReady(): boolean {
|
||||
return this.ready;
|
||||
}
|
||||
|
||||
get helloOk(): HelloOk | null {
|
||||
return this.ready ? this.hello : null;
|
||||
}
|
||||
|
||||
/** Start connecting; reconnects until close(). */
|
||||
start(): void {
|
||||
this.closed = false;
|
||||
this.connect();
|
||||
}
|
||||
|
||||
/** Resolve once the worker has said hello_ok (or reject after timeoutMs). */
|
||||
waitReady(timeoutMs: number): Promise<HelloOk> {
|
||||
if (this.ready && this.hello) return Promise.resolve(this.hello);
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => {
|
||||
this.off("ready", onReady);
|
||||
reject(new Error(`voice worker not ready after ${timeoutMs} ms`));
|
||||
}, timeoutMs);
|
||||
const onReady = (h: HelloOk) => {
|
||||
clearTimeout(t);
|
||||
resolve(h);
|
||||
};
|
||||
this.once("ready", onReady);
|
||||
});
|
||||
}
|
||||
|
||||
close(): void {
|
||||
this.closed = true;
|
||||
if (this.retryTimer) clearTimeout(this.retryTimer);
|
||||
this.retryTimer = null;
|
||||
this.ws?.close(1000);
|
||||
this.ws = null;
|
||||
this.markDisconnected();
|
||||
}
|
||||
|
||||
private connect(): void {
|
||||
if (this.closed) return;
|
||||
const ws = new WebSocket(this.opts.url, { maxPayload: 2 ** 24, perMessageDeflate: false });
|
||||
this.ws = ws;
|
||||
let helloTimer: NodeJS.Timeout | null = null;
|
||||
ws.on("open", () => {
|
||||
ws.send(JSON.stringify({ type: "hello", token: this.opts.token, audio: this.opts.audio, tts: this.opts.tts,
|
||||
stt: this.stt }));
|
||||
helloTimer = setTimeout(() => ws.terminate(), this.opts.helloTimeoutMs ?? 5000);
|
||||
});
|
||||
ws.on("message", (data, isBinary) => {
|
||||
if (isBinary) return this.onBinary(data as Buffer);
|
||||
let msg: Record<string, unknown>;
|
||||
try {
|
||||
msg = JSON.parse(String(data));
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
if (msg.type === "hello_ok") {
|
||||
if (helloTimer) clearTimeout(helloTimer);
|
||||
this.ready = true;
|
||||
this.backoff = this.opts.backoffMs ?? 500;
|
||||
this.hello = msg as unknown as HelloOk;
|
||||
debugLog(`voice worker ready stt=${this.hello.stt_ready} tts=${this.hello.tts_ready}`, "worker");
|
||||
this.emit("ready", this.hello);
|
||||
return;
|
||||
}
|
||||
this.onMessage(msg);
|
||||
});
|
||||
ws.on("close", (code, reason) => {
|
||||
if (helloTimer) clearTimeout(helloTimer);
|
||||
if (this.ws !== ws) return;
|
||||
this.ws = null;
|
||||
if (code === 4400 || code === 4401) {
|
||||
errorLog(`voice worker refused the connection (${code} ${String(reason)})`, "worker");
|
||||
}
|
||||
this.markDisconnected();
|
||||
this.scheduleReconnect();
|
||||
});
|
||||
ws.on("error", (e) => {
|
||||
// "close" follows; connection refused is normal while the worker is still loading.
|
||||
debugLog(`voice worker socket error: ${(e as NodeJS.ErrnoException).code ?? e.message}`, "worker");
|
||||
});
|
||||
}
|
||||
|
||||
private markDisconnected(): void {
|
||||
const was = this.ready;
|
||||
this.ready = false;
|
||||
for (const id of this.openTts) this.emit("tts_done", { id, chunks: 0, cancelled: false, failed: true });
|
||||
this.openTts.clear();
|
||||
if (was) {
|
||||
debugLog(`voice worker disconnected (dropped audio frames so far: ${this.dropped})`, "worker");
|
||||
this.emit("disconnected");
|
||||
}
|
||||
}
|
||||
|
||||
private scheduleReconnect(): void {
|
||||
if (this.closed || this.retryTimer) return;
|
||||
const delay = this.backoff;
|
||||
this.backoff = Math.min(this.backoff * 2, this.opts.maxBackoffMs ?? 5000);
|
||||
this.retryTimer = setTimeout(() => {
|
||||
this.retryTimer = null;
|
||||
this.connect();
|
||||
}, delay);
|
||||
}
|
||||
|
||||
private onBinary(buf: Buffer): void {
|
||||
if (buf[0] !== KIND_TTS) return;
|
||||
const frame = decodeTtsFrame(buf);
|
||||
if (frame && this.openTts.has(frame.id)) this.emit("tts_audio", frame);
|
||||
}
|
||||
|
||||
private onMessage(raw: Record<string, unknown>): void {
|
||||
const { type, ...msg } = raw;
|
||||
switch (type) {
|
||||
case "speech_start":
|
||||
this.emit("speech_start", msg as unknown as SpeechStart);
|
||||
break;
|
||||
case "partial":
|
||||
this.emit("partial", msg as unknown as Partial);
|
||||
break;
|
||||
case "final":
|
||||
this.emit("final", msg as unknown as Final);
|
||||
break;
|
||||
case "tts_done": {
|
||||
const done = msg as unknown as TtsDone;
|
||||
if (this.openTts.delete(done.id)) this.emit("tts_done", done);
|
||||
break;
|
||||
}
|
||||
case "error": {
|
||||
const err = msg as unknown as WorkerError;
|
||||
debugLog(`voice worker error ${err.code}`, "worker");
|
||||
this.emit("worker_error", err);
|
||||
// A request the worker refused outright never gets a tts_done.
|
||||
if (err.code === "tts_unavailable" && err.id && this.openTts.delete(err.id)) {
|
||||
this.emit("tts_done", { id: err.id, chunks: 0, cancelled: false, failed: true });
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private send(data: string | Buffer): boolean {
|
||||
if (!this.ready || !this.ws || this.ws.readyState !== WebSocket.OPEN) return false;
|
||||
this.ws.send(data);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** One frame of a speaker's PCM in the hello "audio" format. Dropped while disconnected. */
|
||||
sendAudio(speaker: string, pcm: Buffer): void {
|
||||
if (!this.send(encodeAudioFrame(speaker, pcm))) this.dropped++;
|
||||
}
|
||||
|
||||
speakingEnd(speaker: string): void {
|
||||
this.send(JSON.stringify({ type: "speaking_end", speaker }));
|
||||
}
|
||||
|
||||
speakerLeave(speaker: string): void {
|
||||
this.send(JSON.stringify({ type: "speaker_leave", speaker }));
|
||||
}
|
||||
|
||||
/** Change STT settings now and for every later connection. */
|
||||
configure(changes: Record<string, unknown>): void {
|
||||
Object.assign(this.stt, changes);
|
||||
this.send(JSON.stringify({ type: "config", ...changes }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Speak `text` as request `id`; with `more` the request stays open for more text.
|
||||
* Returns false (and the request is already failed) when the worker is not connected.
|
||||
*/
|
||||
tts(id: string, text: string, more = false): boolean {
|
||||
const sent = this.send(JSON.stringify({ type: "tts", id, text, more }));
|
||||
if (sent) {
|
||||
this.openTts.add(id);
|
||||
} else if (!this.openTts.has(id)) {
|
||||
queueMicrotask(() => this.emit("tts_done", { id, chunks: 0, cancelled: false, failed: true }));
|
||||
}
|
||||
return sent;
|
||||
}
|
||||
|
||||
ttsCancel(id: string): void {
|
||||
this.send(JSON.stringify({ type: "tts_cancel", id }));
|
||||
}
|
||||
}
|
||||
119
app/src/core/workerProcess.ts
Normal file
119
app/src/core/workerProcess.ts
Normal file
@@ -0,0 +1,119 @@
|
||||
/**
|
||||
* Starts the Python voice worker as a child process and restarts it if it dies.
|
||||
*
|
||||
* Each app launch makes a new random token and hands it over in the environment
|
||||
* (JARVIS_VOICE_TOKEN), never on the command line. The worker listens on 127.0.0.1 only.
|
||||
*/
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { EventEmitter } from "node:events";
|
||||
import { createServer } from "node:net";
|
||||
import { createInterface } from "node:readline";
|
||||
import { debugLog, errorLog } from "./log.js";
|
||||
|
||||
export interface WorkerProcessOptions {
|
||||
python: string;
|
||||
cwd: string;
|
||||
args: string[];
|
||||
/** Fixed port; 0 picks a free one once at start (kept across restarts). */
|
||||
port?: number;
|
||||
/** Extra environment for the worker (merged over the app's). */
|
||||
env?: Record<string, string>;
|
||||
restartDelayMs?: number;
|
||||
maxRestartDelayMs?: number;
|
||||
}
|
||||
|
||||
export async function freePort(): Promise<number> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const srv = createServer();
|
||||
srv.once("error", reject);
|
||||
srv.listen(0, "127.0.0.1", () => {
|
||||
const addr = srv.address();
|
||||
srv.close(() => (typeof addr === "object" && addr ? resolve(addr.port) : reject(new Error("no port"))));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export interface WorkerProcessEvents {
|
||||
/** The worker printed READY: models are loaded and it accepts connections. */
|
||||
ready: [];
|
||||
exit: [{ code: number | null; signal: NodeJS.Signals | null }];
|
||||
}
|
||||
|
||||
export class VoiceWorkerProcess extends EventEmitter<WorkerProcessEvents> {
|
||||
readonly token = randomBytes(32).toString("base64url");
|
||||
private child: ChildProcess | null = null;
|
||||
private stopping = false;
|
||||
private restartTimer: NodeJS.Timeout | null = null;
|
||||
private delay: number;
|
||||
private port = 0;
|
||||
private startedAt = 0;
|
||||
restarts = 0;
|
||||
|
||||
constructor(private readonly opts: WorkerProcessOptions) {
|
||||
super();
|
||||
this.delay = opts.restartDelayMs ?? 1000;
|
||||
}
|
||||
|
||||
get url(): string {
|
||||
return `ws://127.0.0.1:${this.port}`;
|
||||
}
|
||||
|
||||
async start(): Promise<void> {
|
||||
this.stopping = false;
|
||||
this.port = this.opts.port || (await freePort());
|
||||
this.spawnChild();
|
||||
}
|
||||
|
||||
private spawnChild(): void {
|
||||
const args = ["-m", "voice_worker", "--host", "127.0.0.1", "--port", String(this.port), ...this.opts.args];
|
||||
this.startedAt = Date.now();
|
||||
const child = spawn(this.opts.python, args, {
|
||||
cwd: this.opts.cwd,
|
||||
env: { ...process.env, ...this.opts.env, JARVIS_VOICE_TOKEN: this.token, PYTHONUNBUFFERED: "1" },
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
windowsHide: true,
|
||||
});
|
||||
this.child = child;
|
||||
debugLog(`voice worker started pid=${child.pid} port=${this.port}`, "worker");
|
||||
// The worker's own logs hold timings and counts only (voice_worker.spec.md), so they can be passed on.
|
||||
createInterface({ input: child.stdout! }).on("line", (line) => {
|
||||
if (line.startsWith("READY ")) {
|
||||
debugLog(`voice worker READY after ${Date.now() - this.startedAt} ms`, "worker");
|
||||
this.delay = this.opts.restartDelayMs ?? 1000;
|
||||
this.emit("ready");
|
||||
} else debugLog(line, "worker");
|
||||
});
|
||||
createInterface({ input: child.stderr! }).on("line", (line) => debugLog(line, "worker"));
|
||||
child.once("error", (e) => errorLog(`voice worker could not start: ${e.message}`, "worker"));
|
||||
child.once("close", (code, signal) => {
|
||||
if (this.child === child) this.child = null;
|
||||
this.emit("exit", { code, signal });
|
||||
if (this.stopping) return;
|
||||
errorLog(`voice worker exited (code ${code}, signal ${signal}); restarting in ${this.delay} ms`, "worker");
|
||||
this.restartTimer = setTimeout(() => {
|
||||
this.restartTimer = null;
|
||||
this.restarts++;
|
||||
this.spawnChild();
|
||||
}, this.delay);
|
||||
this.delay = Math.min(this.delay * 2, this.opts.maxRestartDelayMs ?? 30_000);
|
||||
});
|
||||
}
|
||||
|
||||
/** Stop the worker and do not restart it. Resolves when it has exited. */
|
||||
async stop(graceMs = 5000): Promise<void> {
|
||||
this.stopping = true;
|
||||
if (this.restartTimer) clearTimeout(this.restartTimer);
|
||||
this.restartTimer = null;
|
||||
const child = this.child;
|
||||
if (!child || child.exitCode !== null || child.signalCode !== null) return;
|
||||
await new Promise<void>((resolve) => {
|
||||
const kill = setTimeout(() => child.kill("SIGKILL"), graceMs);
|
||||
child.once("close", () => {
|
||||
clearTimeout(kill);
|
||||
resolve();
|
||||
});
|
||||
child.kill("SIGTERM");
|
||||
});
|
||||
}
|
||||
}
|
||||
36
app/src/main.ts
Normal file
36
app/src/main.ts
Normal file
@@ -0,0 +1,36 @@
|
||||
/**
|
||||
* Electron main process. The GUI arrives in M5; for now the app runs its services in the background.
|
||||
* The data folder comes from JARVIS_DATA_DIR until the setup wizard picks one (M5).
|
||||
*/
|
||||
import { join } from "node:path";
|
||||
import { app } from "electron";
|
||||
import { errorLog } from "./core/log.js";
|
||||
import { startServices, type Services } from "./services.js";
|
||||
|
||||
const dataDir = process.env.JARVIS_DATA_DIR;
|
||||
// Electron's own profile goes inside the data folder too, not into the user's AppData.
|
||||
if (dataDir) app.setPath("userData", join(dataDir, "cache", "electron"));
|
||||
|
||||
if (!app.requestSingleInstanceLock()) app.quit();
|
||||
|
||||
let services: Services | null = null;
|
||||
|
||||
app.on("window-all-closed", () => {
|
||||
// No windows yet; keep running.
|
||||
});
|
||||
|
||||
app.whenReady().then(async () => {
|
||||
if (!dataDir) throw new Error("JARVIS_DATA_DIR is not set");
|
||||
services = await startServices(dataDir);
|
||||
}).catch((e: Error) => {
|
||||
errorLog(`could not start: ${e.message}`);
|
||||
app.exit(1);
|
||||
});
|
||||
|
||||
let quitting = false;
|
||||
app.on("before-quit", (event) => {
|
||||
if (quitting || !services) return;
|
||||
event.preventDefault();
|
||||
quitting = true;
|
||||
void services.stop().finally(() => app.quit());
|
||||
});
|
||||
57
app/src/services.ts
Normal file
57
app/src/services.ts
Normal file
@@ -0,0 +1,57 @@
|
||||
/**
|
||||
* Starts everything the app runs: the voice worker process, the connection to it, and the Discord
|
||||
* core. Shared by the Electron entry (main.ts) and the plain Node entry (cli.ts).
|
||||
*/
|
||||
import { join } from "node:path";
|
||||
import { DISCORD_PCM, DiscordCore } from "./core/discordCore.js";
|
||||
import { debugLog, errorLog } from "./core/log.js";
|
||||
import { loadSettings } from "./core/settings.js";
|
||||
import { VoiceWorkerClient } from "./core/workerClient.js";
|
||||
import { VoiceWorkerProcess } from "./core/workerProcess.js";
|
||||
|
||||
export interface Services {
|
||||
core: DiscordCore;
|
||||
worker: VoiceWorkerClient;
|
||||
workerProcess: VoiceWorkerProcess;
|
||||
stop(): Promise<void>;
|
||||
}
|
||||
|
||||
export async function startServices(dataDir: string): Promise<Services> {
|
||||
const settings = loadSettings(join(dataDir, "config", "settings.json"));
|
||||
// Until the setup wizard stores it encrypted (M5), the token comes from the environment only.
|
||||
const token = process.env.JARVIS_DISCORD_TOKEN;
|
||||
if (!token) throw new Error("JARVIS_DISCORD_TOKEN is not set");
|
||||
|
||||
const workerProcess = new VoiceWorkerProcess(settings.voiceWorker);
|
||||
await workerProcess.start();
|
||||
const worker = new VoiceWorkerClient({
|
||||
url: workerProcess.url,
|
||||
token: workerProcess.token,
|
||||
audio: DISCORD_PCM,
|
||||
tts: DISCORD_PCM,
|
||||
stt: { ...settings.voice.stt, hotwords: settings.voice.wakeWords },
|
||||
});
|
||||
// The worker loads its models before it listens; connect once it says READY (and after restarts
|
||||
// the client keeps retrying on its own).
|
||||
workerProcess.once("ready", () => worker.start());
|
||||
|
||||
const core = new DiscordCore(settings, token, worker);
|
||||
core.on("final", (f) => debugLog(`heard ${f.speakerId} utt=${f.utt} chars=${f.text.length}`, "app"));
|
||||
try {
|
||||
await core.start();
|
||||
} catch (e) {
|
||||
worker.close();
|
||||
await workerProcess.stop();
|
||||
throw e;
|
||||
}
|
||||
return {
|
||||
core,
|
||||
worker,
|
||||
workerProcess,
|
||||
async stop() {
|
||||
await core.close().catch((e) => errorLog(`discord close: ${(e as Error).message}`));
|
||||
worker.close();
|
||||
await workerProcess.stop();
|
||||
},
|
||||
};
|
||||
}
|
||||
67
app/test/chat.test.ts
Normal file
67
app/test/chat.test.ts
Normal file
@@ -0,0 +1,67 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { ChatAccessError, ChatChannels, hasMessageContentIntent, MESSAGE_LIMIT, splitMessage } from "../src/core/chat.js";
|
||||
|
||||
describe("splitMessage", () => {
|
||||
it("leaves short text alone", () => {
|
||||
expect(splitMessage("안녕하세요")).toEqual(["안녕하세요"]);
|
||||
});
|
||||
|
||||
it("breaks at the last newline, then space, within the limit and loses only the break", () => {
|
||||
const text = "aaaa bbbb\ncccc dddd";
|
||||
const parts = splitMessage(text, 12);
|
||||
expect(parts).toEqual(["aaaa bbbb", "cccc dddd"]);
|
||||
expect(splitMessage("aaaa bbbb cccc", 10)).toEqual(["aaaa bbbb", "cccc"]);
|
||||
});
|
||||
|
||||
it("cuts hard when there is no break, keeping emoji whole", () => {
|
||||
const text = "x".repeat(MESSAGE_LIMIT - 1) + "😀" + "y";
|
||||
const parts = splitMessage(text);
|
||||
expect(parts.every((p) => p.length <= MESSAGE_LIMIT)).toBe(true);
|
||||
expect(parts.join("")).toBe(text);
|
||||
expect(parts[1]!.startsWith("😀")).toBe(true);
|
||||
});
|
||||
|
||||
it("every part fits for long mixed text", () => {
|
||||
const text = Array.from({ length: 900 }, (_, i) => `줄 ${i} 내용입니다`).join("\n");
|
||||
const parts = splitMessage(text);
|
||||
expect(parts.length).toBeGreaterThan(1);
|
||||
expect(parts.every((p) => p.length > 0 && p.length <= MESSAGE_LIMIT)).toBe(true);
|
||||
expect(parts.join("\n")).toBe(text);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ChatChannels", () => {
|
||||
const ch = new ChatChannels([
|
||||
{ id: "111111111111111111", name: "일정", description: "", read: true, send: false },
|
||||
{ id: "222222222222222222", name: "공지", description: "", read: false, send: true },
|
||||
]);
|
||||
|
||||
it("follows each channel's read and send switches", () => {
|
||||
expect(ch.canRead("111111111111111111")).toBe(true);
|
||||
expect(ch.canSend("111111111111111111")).toBe(false);
|
||||
expect(ch.canRead("222222222222222222")).toBe(false);
|
||||
expect(ch.canSend("222222222222222222")).toBe(true);
|
||||
expect(() => ch.assertSend("111111111111111111")).toThrow(ChatAccessError);
|
||||
expect(() => ch.assertRead("222222222222222222")).toThrow(ChatAccessError);
|
||||
});
|
||||
|
||||
it("refuses channels that are not registered", () => {
|
||||
expect(ch.canRead("333333333333333333")).toBe(false);
|
||||
expect(() => ch.assertRead("333333333333333333")).toThrow(/not a registered/);
|
||||
expect(() => ch.assertSend("333333333333333333")).toThrow(/not a registered/);
|
||||
});
|
||||
|
||||
it("needs the message intents only when some channel is readable", () => {
|
||||
expect(ch.anyReadable).toBe(true);
|
||||
expect(new ChatChannels([{ id: "1".repeat(18), name: "", description: "", read: false, send: true }]).anyReadable)
|
||||
.toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("hasMessageContentIntent", () => {
|
||||
it("accepts the full and the limited intent flag", () => {
|
||||
expect(hasMessageContentIntent(1 << 18)).toBe(true);
|
||||
expect(hasMessageContentIntent(1 << 19)).toBe(true);
|
||||
expect(hasMessageContentIntent((1 << 15) | (1 << 13))).toBe(false);
|
||||
});
|
||||
});
|
||||
40
app/test/fixtures/fake_engine_worker.py
vendored
Normal file
40
app/test/fixtures/fake_engine_worker.py
vendored
Normal file
@@ -0,0 +1,40 @@
|
||||
"""The real voice worker server with fake engines, for the app's protocol tests (no GPU, no models).
|
||||
|
||||
The fake VAD reads speech probability off the amplitude; the fake transcriber names a clip by its
|
||||
pitch ("level 0.9" for a 900 Hz tone); the fake TTS makes 10 samples per character.
|
||||
Run with the repo's src/ on PYTHONPATH: python fake_engine_worker.py <port>
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
from voice_worker.server import VoiceWorker
|
||||
|
||||
|
||||
class AmplitudeVad:
|
||||
def prob(self, window):
|
||||
return float(np.max(np.abs(window)))
|
||||
|
||||
|
||||
class LevelTranscriber:
|
||||
def transcribe(self, clips, opts):
|
||||
return [f"level {np.argmax(np.abs(np.fft.rfft(c))) * 16000 / len(c) / 1000:.1f}" for c in clips]
|
||||
|
||||
|
||||
class FakeTts:
|
||||
def synthesize(self, text, rate):
|
||||
return np.full(len(text) * 10, 1000, dtype=np.int16)
|
||||
|
||||
|
||||
async def main(port: int) -> None:
|
||||
worker = VoiceWorker(os.environ["JARVIS_VOICE_TOKEN"], LevelTranscriber(), AmplitudeVad, FakeTts())
|
||||
server = await worker.serve("127.0.0.1", port)
|
||||
print(f"READY 127.0.0.1:{port}", flush=True)
|
||||
await server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main(int(sys.argv[1])))
|
||||
26
app/test/frames.test.ts
Normal file
26
app/test/frames.test.ts
Normal file
@@ -0,0 +1,26 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { decodeTtsFrame, encodeAudioFrame, KIND_AUDIO, KIND_TTS } from "../src/core/frames.js";
|
||||
|
||||
describe("binary frames (voice_worker/protocol.py layout)", () => {
|
||||
it("audio frame is kind, id length, UTF-8 id, then PCM", () => {
|
||||
const pcm = Buffer.from([1, 0, 2, 0]);
|
||||
const f = encodeAudioFrame("유저", pcm);
|
||||
const id = Buffer.from("유저", "utf8");
|
||||
expect([...f]).toEqual([KIND_AUDIO, id.length, ...id, 1, 0, 2, 0]);
|
||||
});
|
||||
|
||||
it("rejects ids that do not fit in a byte", () => {
|
||||
expect(() => encodeAudioFrame("", Buffer.alloc(2))).toThrow();
|
||||
expect(() => encodeAudioFrame("x".repeat(256), Buffer.alloc(2))).toThrow();
|
||||
});
|
||||
|
||||
it("decodes a TTS frame and refuses truncated ones", () => {
|
||||
const id = Buffer.from("r1");
|
||||
const seq = Buffer.alloc(4);
|
||||
seq.writeUInt32LE(7);
|
||||
const frame = Buffer.concat([Buffer.from([KIND_TTS, id.length]), id, seq, Buffer.from([9, 9])]);
|
||||
expect(decodeTtsFrame(frame)).toEqual({ id: "r1", seq: 7, pcm: Buffer.from([9, 9]) });
|
||||
expect(decodeTtsFrame(frame.subarray(0, 5))).toBeNull();
|
||||
expect(decodeTtsFrame(Buffer.from([KIND_AUDIO, 1, 65, 0, 0, 0, 0]))).toBeNull();
|
||||
});
|
||||
});
|
||||
317
app/test/live/discord.live.test.ts
Normal file
317
app/test/live/discord.live.test.ts
Normal file
@@ -0,0 +1,317 @@
|
||||
/**
|
||||
* The Discord voice core end to end on real Discord, with the real voice worker on the GPU.
|
||||
*
|
||||
* Two bots in one voice channel: the "Jarvis" bot runs the core; the "speaker" bot plays recorded
|
||||
* Korean speech into the channel, records what Jarvis says, and writes in the channel's chat.
|
||||
* Opt-in; it refuses to run if a person is in the channel.
|
||||
*
|
||||
* JARVIS_LIVE_DISCORD=1 JARVIS_LIVE_BOT_TOKEN=... JARVIS_LIVE_SPEAKER_TOKEN=... \
|
||||
* JARVIS_LIVE_GUILD=<server id> JARVIS_LIVE_VOICE_CHANNEL=<voice channel id> \
|
||||
* JARVIS_VOICE_PYTHON=<worker venv python> JARVIS_MELO_PYTHON=<MeloTTS venv python> \
|
||||
* npx vitest run test/live
|
||||
*
|
||||
* On Linux, put the pip CUDA libraries on LD_LIBRARY_PATH (nvidia/cublas/lib, nvidia/cudnn/lib).
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { Client, Events, GatewayIntentBits, REST, Routes, type Message, type VoiceBasedChannel } from "discord.js";
|
||||
import {
|
||||
AudioPlayerStatus,
|
||||
createAudioPlayer,
|
||||
createAudioResource,
|
||||
EndBehaviorType,
|
||||
entersState,
|
||||
joinVoiceChannel,
|
||||
StreamType,
|
||||
VoiceConnectionStatus,
|
||||
type VoiceConnection,
|
||||
} from "@discordjs/voice";
|
||||
import OpusScript from "opusscript";
|
||||
import { Readable } from "node:stream";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { DISCORD_PCM, DiscordCore, type HeardFinal } from "../../src/core/discordCore.js";
|
||||
import { parseSettings } from "../../src/core/settings.js";
|
||||
import { VoiceWorkerClient, type Final } from "../../src/core/workerClient.js";
|
||||
import { VoiceWorkerProcess } from "../../src/core/workerProcess.js";
|
||||
import type { PlaybackEnd } from "../../src/core/playback.js";
|
||||
import type { ChatMessage } from "../../src/core/chat.js";
|
||||
|
||||
const env = process.env;
|
||||
const LIVE = env.JARVIS_LIVE_DISCORD === "1";
|
||||
const GUILD = env.JARVIS_LIVE_GUILD!;
|
||||
const VOICE = env.JARVIS_LIVE_VOICE_CHANNEL!;
|
||||
const COMMAND = "jarvis-live-test";
|
||||
const SAMPLES = fileURLToPath(new URL("../../../scripts/m0/assets/ko_samples/", import.meta.url));
|
||||
const SRC = fileURLToPath(new URL("../../../src", import.meta.url));
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
async function until(pred: () => boolean, ms: number, what: string) {
|
||||
const end = Date.now() + ms;
|
||||
while (!pred()) {
|
||||
if (Date.now() > end) throw new Error(`timed out waiting for ${what}`);
|
||||
await sleep(25);
|
||||
}
|
||||
}
|
||||
|
||||
function cer(ref: string, hyp: string): number {
|
||||
const norm = (s: string) => [...s.replace(/[^\p{L}\p{N}]+/gu, "")];
|
||||
const a = norm(ref);
|
||||
const b = norm(hyp);
|
||||
let prev = Array.from({ length: b.length + 1 }, (_, j) => j);
|
||||
for (let i = 1; i <= a.length; i++) {
|
||||
const cur = [i];
|
||||
for (let j = 1; j <= b.length; j++) {
|
||||
cur.push(Math.min(prev[j]! + 1, cur[j - 1]! + 1, prev[j - 1]! + (a[i - 1] === b[j - 1] ? 0 : 1)));
|
||||
}
|
||||
prev = cur;
|
||||
}
|
||||
return prev[b.length]! / Math.max(1, a.length);
|
||||
}
|
||||
|
||||
/** Seconds into 48 kHz stereo PCM where the last sound above -34 dBFS ends (the end of speech). */
|
||||
function speechEndSeconds(pcm: Buffer): number {
|
||||
for (let k = pcm.length / 4 - 1; k >= 0; k--) if (Math.abs(pcm.readInt16LE(k * 4)) > 655) return (k + 1) / 48000;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** A 16 kHz mono PCM16 WAV as 48 kHz stereo PCM16 (Discord's format). */
|
||||
function wavTo48kStereo(path: string): Buffer {
|
||||
const wav = readFileSync(path);
|
||||
let off = 12;
|
||||
let data: Buffer | null = null;
|
||||
while (off + 8 <= wav.length) {
|
||||
const id = wav.toString("ascii", off, off + 4);
|
||||
const size = wav.readUInt32LE(off + 4);
|
||||
if (id === "data") data = wav.subarray(off + 8, off + 8 + size);
|
||||
off += 8 + size + (size % 2);
|
||||
}
|
||||
if (!data) throw new Error(`no data chunk in ${path}`);
|
||||
const n = data.length / 2;
|
||||
const out = Buffer.alloc(n * 3 * 4);
|
||||
for (let k = 0; k < n * 3; k++) {
|
||||
const pos = k / 3;
|
||||
const i = Math.floor(pos);
|
||||
const a = data.readInt16LE(i * 2);
|
||||
const b = i + 1 < n ? data.readInt16LE((i + 1) * 2) : a;
|
||||
const v = Math.round(a + (b - a) * (pos - i));
|
||||
out.writeInt16LE(v, k * 4);
|
||||
out.writeInt16LE(v, k * 4 + 2);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe.skipIf(!LIVE)("Discord voice core on real Discord", () => {
|
||||
let workerProcess: VoiceWorkerProcess;
|
||||
let worker: VoiceWorkerClient;
|
||||
let core: DiscordCore;
|
||||
let speaker: Client;
|
||||
let speakerVoice: VoiceConnection;
|
||||
let jarvisId = "";
|
||||
const finals: HeardFinal[] = [];
|
||||
const speakEnds: PlaybackEnd[] = [];
|
||||
const chats: ChatMessage[] = [];
|
||||
/** Jarvis's voice as the speaker bot hears it: decoded PCM and packet times. */
|
||||
const heard = { pcm: [] as Buffer[], lastPacketAt: 0, packets: 0 };
|
||||
|
||||
beforeAll(async () => {
|
||||
// The speaker bot first: it checks the channel is free of people before anything joins.
|
||||
speaker = new Client({ intents: [GatewayIntentBits.Guilds, GatewayIntentBits.GuildVoiceStates,
|
||||
GatewayIntentBits.GuildMessages, GatewayIntentBits.MessageContent] });
|
||||
await speaker.login(env.JARVIS_LIVE_SPEAKER_TOKEN);
|
||||
await new Promise<void>((r) => (speaker.isReady() ? r() : speaker.once(Events.ClientReady, () => r())));
|
||||
const guild = await speaker.guilds.fetch(GUILD);
|
||||
const people = guild.voiceStates.cache.filter((v) => v.channelId === VOICE && !v.member?.user.bot);
|
||||
if (people.size) throw new Error(`refusing to test: ${people.size} people are in the voice channel`);
|
||||
// Either bot already in a voice channel of this server is in use elsewhere; joining would move it.
|
||||
const jarvisApp = (await new REST({ version: "10" }).setToken(env.JARVIS_LIVE_BOT_TOKEN!)
|
||||
.get(Routes.currentApplication())) as { id: string };
|
||||
for (const id of [speaker.user!.id, jarvisApp.id]) {
|
||||
if (guild.voiceStates.cache.get(id)?.channelId) throw new Error(`refusing to test: bot ${id} is in a voice channel`);
|
||||
}
|
||||
|
||||
workerProcess = new VoiceWorkerProcess({ python: env.JARVIS_VOICE_PYTHON!, cwd: SRC,
|
||||
args: env.JARVIS_MELO_PYTHON ? ["--melo-python", env.JARVIS_MELO_PYTHON] : [] });
|
||||
const workerReady = new Promise<void>((r) => workerProcess.once("ready", () => r()));
|
||||
await workerProcess.start();
|
||||
await workerReady;
|
||||
worker = new VoiceWorkerClient({ url: workerProcess.url, token: workerProcess.token, audio: DISCORD_PCM,
|
||||
tts: DISCORD_PCM, stt: { hotwords: ["자비스"] } });
|
||||
worker.start();
|
||||
await worker.waitReady(10_000);
|
||||
|
||||
const settings = parseSettings({
|
||||
discord: { guildId: GUILD, voiceChannelId: VOICE, commandName: COMMAND },
|
||||
// The "person" here is a bot, so bots must be heard.
|
||||
voice: { listenToBots: true },
|
||||
// The voice channel's own chat stands in for a registered chat channel.
|
||||
chatChannels: [{ id: VOICE, name: "테스트", description: "라이브 테스트", read: true, send: true }],
|
||||
});
|
||||
core = new DiscordCore(settings, env.JARVIS_LIVE_BOT_TOKEN!, worker);
|
||||
core.on("final", (f) => finals.push(f));
|
||||
core.on("speak_end", (e) => speakEnds.push(e));
|
||||
core.on("chat", (c) => chats.push(c));
|
||||
core.on("ready", (r) => (jarvisId = r.userId));
|
||||
await core.start(); // logs in, registers the command, auto-joins VOICE, posts the notice
|
||||
expect(core.voiceChannelId).toBe(VOICE);
|
||||
|
||||
const channel = (await speaker.channels.fetch(VOICE)) as VoiceBasedChannel;
|
||||
speakerVoice = joinVoiceChannel({ channelId: VOICE, guildId: GUILD, adapterCreator: channel.guild.voiceAdapterCreator,
|
||||
selfDeaf: false, group: "speaker" });
|
||||
await entersState(speakerVoice, VoiceConnectionStatus.Ready, 20_000);
|
||||
speakerVoice.receiver.speaking.on("start", (userId) => {
|
||||
if (userId !== jarvisId) return;
|
||||
const stream = speakerVoice.receiver.subscribe(userId, { end: { behavior: EndBehaviorType.Manual } });
|
||||
if (stream.listenerCount("data")) return;
|
||||
const dec = new OpusScript(48000, 2, OpusScript.Application.VOIP);
|
||||
stream.on("data", (p: Buffer) => {
|
||||
heard.packets++;
|
||||
heard.lastPacketAt = Date.now();
|
||||
try {
|
||||
heard.pcm.push(Buffer.from(dec.decode(p)));
|
||||
} catch { /* corrupt packet */ }
|
||||
});
|
||||
});
|
||||
}, 240_000);
|
||||
|
||||
afterAll(async () => {
|
||||
try {
|
||||
const rest = new REST({ version: "10" }).setToken(env.JARVIS_LIVE_BOT_TOKEN!);
|
||||
const app = (await rest.get(Routes.currentApplication())) as { id: string };
|
||||
const cmds = (await rest.get(Routes.applicationGuildCommands(app.id, GUILD))) as { id: string; name: string }[];
|
||||
for (const c of cmds.filter((c) => c.name === COMMAND)) {
|
||||
await rest.delete(Routes.applicationGuildCommand(app.id, GUILD, c.id));
|
||||
}
|
||||
} catch { /* best effort */ }
|
||||
speakerVoice?.destroy();
|
||||
await core?.close();
|
||||
await speaker?.destroy();
|
||||
worker?.close();
|
||||
await workerProcess?.stop();
|
||||
}, 60_000);
|
||||
|
||||
it("registered the slash command in the server", async () => {
|
||||
const rest = new REST({ version: "10" }).setToken(env.JARVIS_LIVE_BOT_TOKEN!);
|
||||
const app = (await rest.get(Routes.currentApplication())) as { id: string };
|
||||
const cmds = (await rest.get(Routes.applicationGuildCommands(app.id, GUILD))) as
|
||||
{ name: string; options: { name: string }[] }[];
|
||||
const cmd = cmds.find((c) => c.name === COMMAND);
|
||||
expect(cmd?.options.map((o) => o.name).sort()).toEqual(["join", "leave", "stop"]);
|
||||
});
|
||||
|
||||
it("hears each recorded sentence played into the channel as a final transcript", async () => {
|
||||
const refs = JSON.parse(readFileSync(`${SAMPLES}refs.json`, "utf8")).samples as { file: string; text: string }[];
|
||||
const player = createAudioPlayer();
|
||||
speakerVoice.subscribe(player);
|
||||
const speakerId = speaker.user!.id;
|
||||
const results: { ref: string; hyp: string; cer: number; speechEndToFinalMs: number; endReason: string }[] = [];
|
||||
for (const ref of refs.slice(0, 5)) {
|
||||
const before = finals.length;
|
||||
const pcm = wavTo48kStereo(SAMPLES + ref.file);
|
||||
player.play(createAudioResource(Readable.from([pcm]), { inputType: StreamType.Raw }));
|
||||
await entersState(player, AudioPlayerStatus.Playing, 5000);
|
||||
const speechEndsAt = Date.now() + speechEndSeconds(pcm) * 1000;
|
||||
let finalAt = 0;
|
||||
await until(() => {
|
||||
const ok = finals.slice(before).some((f) => f.speakerId === speakerId);
|
||||
if (ok && !finalAt) finalAt = Date.now();
|
||||
return ok;
|
||||
}, 20_000, `final for ${ref.file}`);
|
||||
await entersState(player, AudioPlayerStatus.Idle, 20_000);
|
||||
await sleep(800);
|
||||
const mine = finals.slice(before).filter((f) => f.speakerId === speakerId);
|
||||
const hyp = mine.map((f) => f.text).join(" ");
|
||||
results.push({ ref: ref.text, hyp, cer: cer(ref.text, hyp), speechEndToFinalMs: finalAt - speechEndsAt,
|
||||
endReason: mine.map((f) => f.endReason).join(",") });
|
||||
expect(mine[0]!.speakerName.length).toBeGreaterThan(0);
|
||||
}
|
||||
console.log("live STT results", JSON.stringify(results, null, 1));
|
||||
const mean = results.reduce((s, r) => s + r.cer, 0) / results.length;
|
||||
expect(results.every((r) => r.hyp.length > 0)).toBe(true);
|
||||
expect(mean).toBeLessThan(0.15);
|
||||
}, 180_000);
|
||||
|
||||
it("speaks a reply the other side hears, and the STT reads it back", async () => {
|
||||
const text = "안녕하세요, 저는 자비스입니다. 디스코드 음성 연결을 확인하고 있어요.";
|
||||
heard.pcm.length = 0;
|
||||
const before = speakEnds.length;
|
||||
const id = core.say(text);
|
||||
expect(id).not.toBeNull();
|
||||
await until(() => speakEnds.length > before, 30_000, "speech to end");
|
||||
expect(speakEnds.at(-1)).toMatchObject({ id, cancelled: false, failed: false });
|
||||
await sleep(500);
|
||||
const pcm = Buffer.concat(heard.pcm);
|
||||
const seconds = pcm.length / (48000 * 4);
|
||||
expect(seconds).toBeGreaterThan(2);
|
||||
// Read Jarvis's voice back through the STT on a second worker session.
|
||||
const echo = new VoiceWorkerClient({ url: workerProcess.url, token: workerProcess.token, audio: DISCORD_PCM,
|
||||
tts: DISCORD_PCM, stt: { partials: false } });
|
||||
echo.start();
|
||||
await echo.waitReady(5000);
|
||||
const got: Final[] = [];
|
||||
echo.on("final", (f) => got.push(f));
|
||||
const frame = 3840;
|
||||
for (let k = 0; k < pcm.length; k += frame) echo.sendAudio("echo", pcm.subarray(k, k + frame));
|
||||
for (let k = 0; k < 25; k++) echo.sendAudio("echo", Buffer.alloc(frame)); // 0.5 s of silence
|
||||
echo.speakingEnd("echo");
|
||||
await sleep(3000);
|
||||
echo.close();
|
||||
const hyp = got.map((f) => f.text).join(" ");
|
||||
console.log("live TTS read back", JSON.stringify({ seconds, hyp, cer: cer(text, hyp) }));
|
||||
expect(cer(text, hyp)).toBeLessThan(0.2);
|
||||
}, 90_000);
|
||||
|
||||
it("stop silences it within a few frames and cancels the rest", async () => {
|
||||
const text = "이 문장은 정지 명령을 시험하려고 길게 말하는 첫 번째 문장입니다. "
|
||||
+ "두 번째 문장도 꽤 길게 이어집니다. 세 번째 문장까지 들리면 정지가 안 된 것입니다. "
|
||||
+ "네 번째 문장은 절대 들리면 안 됩니다.";
|
||||
const before = speakEnds.length;
|
||||
let started = false;
|
||||
core.once("speak_start", () => (started = true));
|
||||
const id = core.say(text);
|
||||
await until(() => started, 30_000, "speech to start");
|
||||
await sleep(1500);
|
||||
const stopAt = Date.now();
|
||||
expect(core.stop()).toBe(1);
|
||||
await sleep(1500);
|
||||
expect(speakEnds.slice(before)).toEqual([expect.objectContaining({ id, cancelled: true })]);
|
||||
const tail = heard.lastPacketAt - stopAt;
|
||||
console.log("live stop", JSON.stringify({ lastPacketAfterStopMs: tail }));
|
||||
// The player sends no padding after a forced stop; allow for network delivery.
|
||||
expect(tail).toBeLessThan(400);
|
||||
expect(core.speaking).toBe(false);
|
||||
}, 60_000);
|
||||
|
||||
it("reads the registered chat channel live and in history, and sends to it without pings", async () => {
|
||||
const channel = (await speaker.channels.fetch(VOICE)) as VoiceBasedChannel;
|
||||
const marker = `라이브 테스트 메시지 ${Date.now()}`;
|
||||
await channel.send(marker);
|
||||
await until(() => chats.some((c) => c.content === marker), 10_000, "chat event");
|
||||
expect(chats.find((c) => c.content === marker)).toMatchObject({ channelId: VOICE, authorId: speaker.user!.id,
|
||||
authorIsBot: true, fromSelf: false });
|
||||
|
||||
const history = await core.readChat(VOICE, 20);
|
||||
expect(history.some((m) => m.content === marker && !m.fromSelf)).toBe(true);
|
||||
// The transcription notice from joining is there, written by Jarvis.
|
||||
expect(history.some((m) => m.fromSelf && m.content.includes("받아 적습니다"))).toBe(true);
|
||||
|
||||
const seen: Message[] = [];
|
||||
speaker.on(Events.MessageCreate, (m) => void (m.author.id === jarvisId && seen.push(m)));
|
||||
const reply = `자비스 답장 @everyone ${Date.now()}`;
|
||||
const ids = await core.sendChat(VOICE, reply);
|
||||
await until(() => seen.some((m) => ids.includes(m.id)), 10_000, "sent message");
|
||||
const sent = seen.find((m) => ids.includes(m.id))!;
|
||||
expect(sent.content).toBe(reply);
|
||||
expect(sent.mentions.everyone).toBe(false);
|
||||
// Our own message does not come back as a chat event.
|
||||
await sleep(500);
|
||||
expect(chats.some((c) => c.content === reply)).toBe(false);
|
||||
}, 60_000);
|
||||
|
||||
it("leaves the voice channel", async () => {
|
||||
const guild = await speaker.guilds.fetch(GUILD);
|
||||
expect(core.leave()).toBe(true);
|
||||
await until(() => guild.voiceStates.cache.get(jarvisId)?.channelId == null, 10_000, "Jarvis to leave");
|
||||
expect(core.voiceChannelId).toBeNull();
|
||||
expect(core.say("아무도 못 듣는 말")).toBeNull();
|
||||
}, 30_000);
|
||||
});
|
||||
121
app/test/playback.test.ts
Normal file
121
app/test/playback.test.ts
Normal file
@@ -0,0 +1,121 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { Playback, type PlaybackEnd } from "../src/core/playback.js";
|
||||
|
||||
/** A fake worker and speaker: records what was asked of them and what was played. */
|
||||
function setup() {
|
||||
const sent: { id: string; text: string; more: boolean }[] = [];
|
||||
const cancelled: string[] = [];
|
||||
const played: string[] = [];
|
||||
let stops = 0;
|
||||
const pb = new Playback(
|
||||
{ tts: (id, text, more = false) => (sent.push({ id, text, more }), true), ttsCancel: (id) => void cancelled.push(id) },
|
||||
{ play: (pcm) => void played.push(pcm.toString()), stop: () => void stops++ },
|
||||
);
|
||||
const ends: PlaybackEnd[] = [];
|
||||
const starts: string[] = [];
|
||||
pb.on("end", (e) => ends.push(e));
|
||||
pb.on("start", (id) => starts.push(id));
|
||||
return { pb, sent, cancelled, played, ends, starts, stops: () => stops };
|
||||
}
|
||||
|
||||
const pcm = (s: string) => Buffer.from(s);
|
||||
|
||||
describe("Playback", () => {
|
||||
it("plays the first sentence as soon as it arrives, then the next when the clip ends", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("하나. 둘.");
|
||||
expect(t.sent).toEqual([{ id, text: "하나. 둘.", more: false }]);
|
||||
t.pb.audio(id, 0, pcm("s0"));
|
||||
expect(t.played).toEqual(["s0"]);
|
||||
expect(t.starts).toEqual([id]);
|
||||
t.pb.audio(id, 1, pcm("s1"));
|
||||
expect(t.played).toEqual(["s0"]); // still playing s0
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["s0", "s1"]);
|
||||
t.pb.done({ id, chunks: 2, cancelled: false, failed: false });
|
||||
expect(t.ends).toEqual([]); // s1 is still playing
|
||||
t.pb.clipEnded();
|
||||
expect(t.ends).toEqual([{ id, played: 2, cancelled: false, failed: false }]);
|
||||
expect(t.pb.busy).toBe(false);
|
||||
});
|
||||
|
||||
it("plays requests in the order they were made", () => {
|
||||
const t = setup();
|
||||
const a = t.pb.say("a");
|
||||
const b = t.pb.say("b");
|
||||
t.pb.audio(b, 0, pcm("b0")); // b's audio arrives first
|
||||
expect(t.played).toEqual([]);
|
||||
t.pb.audio(a, 0, pcm("a0"));
|
||||
t.pb.done({ id: a, chunks: 1, cancelled: false, failed: false });
|
||||
t.pb.done({ id: b, chunks: 1, cancelled: false, failed: false });
|
||||
t.pb.clipEnded();
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["a0", "b0"]);
|
||||
expect(t.ends.map((e) => e.id)).toEqual([a, b]);
|
||||
});
|
||||
|
||||
it("keeps sentences in order even when they arrive out of order", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("x", true);
|
||||
t.pb.audio(id, 1, pcm("s1"));
|
||||
expect(t.played).toEqual([]);
|
||||
t.pb.audio(id, 0, pcm("s0"));
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["s0", "s1"]);
|
||||
});
|
||||
|
||||
it("skips a sentence that failed and still plays the rest", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("a. b. c.");
|
||||
t.pb.audio(id, 0, pcm("s0"));
|
||||
t.pb.audio(id, 2, pcm("s2"));
|
||||
t.pb.done({ id, chunks: 2, cancelled: false, failed: true });
|
||||
t.pb.clipEnded();
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["s0", "s2"]);
|
||||
expect(t.ends).toEqual([{ id, played: 2, cancelled: false, failed: true }]);
|
||||
});
|
||||
|
||||
it("stop silences the clip, cancels unfinished requests and drops the queue", () => {
|
||||
const t = setup();
|
||||
const a = t.pb.say("a", true);
|
||||
const b = t.pb.say("b");
|
||||
t.pb.audio(a, 0, pcm("a0"));
|
||||
t.pb.done({ id: b, chunks: 0, cancelled: false, failed: false });
|
||||
expect(t.pb.stop()).toBe(2);
|
||||
expect(t.stops()).toBe(1);
|
||||
expect(t.cancelled).toEqual([a]); // b was already done at the worker
|
||||
expect(t.ends).toEqual([
|
||||
{ id: a, played: 1, cancelled: true, failed: false },
|
||||
{ id: b, played: 0, cancelled: true, failed: false },
|
||||
]);
|
||||
// Late audio for a stopped request is ignored.
|
||||
t.pb.audio(a, 1, pcm("a1"));
|
||||
t.pb.clipEnded();
|
||||
expect(t.played).toEqual(["a0"]);
|
||||
expect(t.pb.busy).toBe(false);
|
||||
});
|
||||
|
||||
it("stop with nothing to say reports zero and touches nothing", () => {
|
||||
const t = setup();
|
||||
expect(t.pb.stop()).toBe(0);
|
||||
expect(t.stops()).toBe(0);
|
||||
});
|
||||
|
||||
it("text can be added to an open request", () => {
|
||||
const t = setup();
|
||||
const id = t.pb.say("첫 문장.", true);
|
||||
t.pb.continueText(id, " 둘째 문장.", false);
|
||||
expect(t.sent.map((s) => [s.text, s.more])).toEqual([["첫 문장.", true], [" 둘째 문장.", false]]);
|
||||
});
|
||||
|
||||
it("a request the worker never accepted ends as failed without blocking the next", () => {
|
||||
const t = setup();
|
||||
const a = t.pb.say("a");
|
||||
const b = t.pb.say("b");
|
||||
t.pb.done({ id: a, chunks: 0, cancelled: false, failed: true });
|
||||
t.pb.audio(b, 0, pcm("b0"));
|
||||
expect(t.played).toEqual(["b0"]);
|
||||
expect(t.ends[0]).toEqual({ id: a, played: 0, cancelled: false, failed: true });
|
||||
});
|
||||
});
|
||||
61
app/test/settings.test.ts
Normal file
61
app/test/settings.test.ts
Normal file
@@ -0,0 +1,61 @@
|
||||
import { mkdtempSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { defaultSettings, loadSettings, parseSettings, SettingsError } from "../src/core/settings.js";
|
||||
|
||||
const ID = "123456789012345678";
|
||||
|
||||
describe("settings", () => {
|
||||
it("an empty file gives the defaults", () => {
|
||||
expect(parseSettings({})).toEqual(defaultSettings());
|
||||
});
|
||||
|
||||
it("a missing file gives the defaults", () => {
|
||||
const dir = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-settings-"));
|
||||
expect(loadSettings(join(dir, "nope.json"))).toEqual(defaultSettings());
|
||||
});
|
||||
|
||||
it("reads a full file, BOM included", () => {
|
||||
const dir = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-settings-"));
|
||||
const path = join(dir, "settings.json");
|
||||
writeFileSync(path, "" + JSON.stringify({
|
||||
discord: { guildId: ID, voiceChannelId: ID, commandName: "jarvis", ephemeralReplies: false },
|
||||
voice: { maxSpeakers: 3, listenToBots: true, joinNotice: "알림", wakeWords: [" 자비스 ", "", "javis"],
|
||||
stt: { end_silence_ms: 250 } },
|
||||
chatChannels: [{ id: ID, name: "일정", description: "일정 올리는 곳", send: false }],
|
||||
voiceWorker: { python: "/venv/bin/python", cwd: "/repo/src", args: ["--melo-python", "/m/python"] },
|
||||
somethingNew: 1,
|
||||
}));
|
||||
const s = loadSettings(path);
|
||||
expect(s.discord).toEqual({ guildId: ID, voiceChannelId: ID, commandName: "jarvis", ephemeralReplies: false });
|
||||
expect(s.voice).toEqual({ maxSpeakers: 3, listenToBots: true, joinNotice: "알림", wakeWords: ["자비스", "javis"],
|
||||
stt: { end_silence_ms: 250 } });
|
||||
expect(s.chatChannels).toEqual([{ id: ID, name: "일정", description: "일정 올리는 곳", read: true, send: false }]);
|
||||
expect(s.voiceWorker.args).toEqual(["--melo-python", "/m/python"]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[{ discord: { guildId: "abc" } }, /discord.guildId/],
|
||||
[{ discord: { commandName: "Jarvis" } }, /commandName/],
|
||||
[{ discord: { commandName: "두 단어" } }, /commandName/],
|
||||
[{ voice: { maxSpeakers: 11 } }, /maxSpeakers/],
|
||||
[{ voice: { maxSpeakers: 0 } }, /maxSpeakers/],
|
||||
[{ voice: { joinNotice: " " } }, /joinNotice/],
|
||||
[{ chatChannels: [{ id: ID }, { id: ID }] }, /twice/],
|
||||
[{ chatChannels: [{ id: "1" }] }, /chatChannels\[0\].id/],
|
||||
[{ chatChannels: {} }, /chatChannels/],
|
||||
[{ voice: [] }, /voice/],
|
||||
[[], /object/],
|
||||
])("refuses %j", (raw, message) => {
|
||||
expect(() => parseSettings(raw)).toThrow(SettingsError);
|
||||
expect(() => parseSettings(raw)).toThrow(message);
|
||||
});
|
||||
|
||||
it("reports invalid JSON with the path", () => {
|
||||
const dir = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-settings-"));
|
||||
const path = join(dir, "settings.json");
|
||||
writeFileSync(path, "{ nope");
|
||||
expect(() => loadSettings(path)).toThrow(/not valid JSON/);
|
||||
});
|
||||
});
|
||||
24
app/test/speakers.test.ts
Normal file
24
app/test/speakers.test.ts
Normal file
@@ -0,0 +1,24 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { SpeakerGate } from "../src/core/speakers.js";
|
||||
|
||||
describe("SpeakerGate", () => {
|
||||
it("hears up to the limit and lets the next person in once someone stops", () => {
|
||||
const g = new SpeakerGate(2);
|
||||
expect(g.start("a")).toBe(true);
|
||||
expect(g.start("b")).toBe(true);
|
||||
expect(g.start("c")).toBe(false);
|
||||
expect(g.isActive("c")).toBe(false);
|
||||
expect(g.end("a")).toBe(true);
|
||||
expect(g.start("c")).toBe(true);
|
||||
expect(g.count).toBe(2);
|
||||
});
|
||||
|
||||
it("a speaker already heard stays heard; ending someone not heard is a no-op", () => {
|
||||
const g = new SpeakerGate(1);
|
||||
expect(g.start("a")).toBe(true);
|
||||
expect(g.start("a")).toBe(true);
|
||||
expect(g.end("z")).toBe(false);
|
||||
expect(g.end("a")).toBe(true);
|
||||
expect(g.end("a")).toBe(false);
|
||||
});
|
||||
});
|
||||
5
app/test/tsconfig.json
Normal file
5
app/test/tsconfig.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"extends": "../tsconfig.json",
|
||||
"compilerOptions": { "noEmit": true, "rootDir": ".." },
|
||||
"include": ["../src", "."]
|
||||
}
|
||||
171
app/test/workerClient.test.ts
Normal file
171
app/test/workerClient.test.ts
Normal file
@@ -0,0 +1,171 @@
|
||||
import { once } from "node:events";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { WebSocketServer, type WebSocket } from "ws";
|
||||
import { decodeTtsFrame, encodeAudioFrame, KIND_TTS } from "../src/core/frames.js";
|
||||
import { VoiceWorkerClient, type TtsDone } from "../src/core/workerClient.js";
|
||||
|
||||
const TOKEN = "t0ken";
|
||||
const FMT = { rate: 48000, channels: 2 } as const;
|
||||
|
||||
/** A stand-in voice worker: checks the hello and records what the client sends. */
|
||||
async function fakeWorker(port = 0) {
|
||||
const wss = new WebSocketServer({ host: "127.0.0.1", port });
|
||||
await once(wss, "listening");
|
||||
const received: (Record<string, unknown> | Buffer)[] = [];
|
||||
const sockets: WebSocket[] = [];
|
||||
wss.on("connection", (ws) => {
|
||||
sockets.push(ws);
|
||||
ws.on("message", (data, isBinary) => {
|
||||
if (isBinary) return void received.push(data as Buffer);
|
||||
const msg = JSON.parse(String(data));
|
||||
received.push(msg);
|
||||
if (msg.type === "hello") {
|
||||
if (msg.token !== TOKEN) return ws.close(4401, "bad token");
|
||||
ws.send(JSON.stringify({ type: "hello_ok", stt_ready: true, tts_ready: true, stt: msg.stt }));
|
||||
}
|
||||
});
|
||||
});
|
||||
const addr = wss.address() as { port: number };
|
||||
return { wss, received, sockets, url: `ws://127.0.0.1:${addr.port}`, port: addr.port };
|
||||
}
|
||||
|
||||
const cleanups: (() => void | Promise<void>)[] = [];
|
||||
afterEach(async () => {
|
||||
for (const c of cleanups.splice(0)) await c();
|
||||
});
|
||||
|
||||
function client(url: string, extra: Partial<ConstructorParameters<typeof VoiceWorkerClient>[0]> = {}) {
|
||||
const c = new VoiceWorkerClient({ url, token: TOKEN, audio: FMT, tts: FMT, backoffMs: 20, maxBackoffMs: 50, ...extra });
|
||||
cleanups.push(() => c.close());
|
||||
return c;
|
||||
}
|
||||
|
||||
const until = async (pred: () => boolean, ms = 3000) => {
|
||||
const end = Date.now() + ms;
|
||||
while (!pred()) {
|
||||
if (Date.now() > end) throw new Error("timed out");
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
}
|
||||
};
|
||||
|
||||
describe("VoiceWorkerClient", () => {
|
||||
it("says hello with the token, formats and STT settings, then streams audio frames", async () => {
|
||||
const w = await fakeWorker();
|
||||
cleanups.push(() => void w.wss.close());
|
||||
const c = client(w.url, { stt: { hotwords: ["자비스"] } });
|
||||
c.start();
|
||||
const ok = await c.waitReady(2000);
|
||||
expect(ok.stt).toEqual({ hotwords: ["자비스"] });
|
||||
expect(w.received[0]).toEqual({ type: "hello", token: TOKEN, audio: FMT, tts: FMT, stt: { hotwords: ["자비스"] } });
|
||||
const pcm = Buffer.alloc(3840, 1);
|
||||
c.sendAudio("123", pcm);
|
||||
c.speakingEnd("123");
|
||||
c.speakerLeave("123");
|
||||
await until(() => w.received.length >= 4);
|
||||
expect(w.received[1]).toEqual(encodeAudioFrame("123", pcm));
|
||||
expect(w.received.slice(2)).toEqual([{ type: "speaking_end", speaker: "123" }, { type: "speaker_leave", speaker: "123" }]);
|
||||
});
|
||||
|
||||
it("passes worker events on and only TTS audio of requests it opened", async () => {
|
||||
const w = await fakeWorker();
|
||||
cleanups.push(() => void w.wss.close());
|
||||
const c = client(w.url);
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
const finals: unknown[] = [];
|
||||
const audio: { id: string; seq: number }[] = [];
|
||||
const done: TtsDone[] = [];
|
||||
c.on("final", (f) => finals.push(f));
|
||||
c.on("tts_audio", (a) => audio.push({ id: a.id, seq: a.seq }));
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
expect(c.tts("r1", "안녕하세요.")).toBe(true);
|
||||
await until(() => w.received.some((m) => !Buffer.isBuffer(m) && m.type === "tts"));
|
||||
const ws = w.sockets[0]!;
|
||||
const frame = (id: string, seq: number) => {
|
||||
const s = Buffer.alloc(4);
|
||||
s.writeUInt32LE(seq);
|
||||
return Buffer.concat([Buffer.from([KIND_TTS, id.length]), Buffer.from(id), s, Buffer.alloc(8)]);
|
||||
};
|
||||
ws.send(frame("r1", 0));
|
||||
ws.send(frame("zz", 0)); // not ours
|
||||
ws.send(JSON.stringify({ type: "final", speaker: "1", utt: 1, text: "네", end_reason: "silence", audio_ms: 900,
|
||||
wait_ms: 100, stt_ms: 80 }));
|
||||
ws.send(JSON.stringify({ type: "tts_done", id: "r1", chunks: 1, cancelled: false, failed: false }));
|
||||
await until(() => done.length === 1 && finals.length === 1);
|
||||
expect(audio).toEqual([{ id: "r1", seq: 0 }]);
|
||||
expect(decodeTtsFrame(frame("r1", 0))?.pcm.length).toBe(8);
|
||||
expect(done[0]).toEqual({ id: "r1", chunks: 1, cancelled: false, failed: false });
|
||||
});
|
||||
|
||||
it("a refused TTS request (no engine) ends as failed", async () => {
|
||||
const w = await fakeWorker();
|
||||
cleanups.push(() => void w.wss.close());
|
||||
const c = client(w.url);
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
const done: TtsDone[] = [];
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
c.tts("r9", "x");
|
||||
await until(() => w.received.length >= 2);
|
||||
w.sockets[0]!.send(JSON.stringify({ type: "error", code: "tts_unavailable", message: "no", id: "r9" }));
|
||||
await until(() => done.length === 1);
|
||||
expect(done[0]).toMatchObject({ id: "r9", failed: true });
|
||||
});
|
||||
|
||||
it("reconnects after the worker restarts and fails the TTS that was in flight", async () => {
|
||||
let w = await fakeWorker();
|
||||
const port = w.port;
|
||||
const c = client(w.url);
|
||||
let readies = 0;
|
||||
let disconnects = 0;
|
||||
c.on("ready", () => readies++);
|
||||
c.on("disconnected", () => disconnects++);
|
||||
const done: TtsDone[] = [];
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
c.tts("r1", "길게 말하는 중");
|
||||
await until(() => w.received.length >= 2);
|
||||
// The worker dies...
|
||||
for (const s of w.sockets) s.terminate();
|
||||
await new Promise<void>((r) => w.wss.close(() => r()));
|
||||
await until(() => disconnects === 1);
|
||||
expect(done).toEqual([{ id: "r1", chunks: 0, cancelled: false, failed: true }]);
|
||||
expect(c.isReady).toBe(false);
|
||||
// Audio while it is down is dropped, not queued.
|
||||
c.sendAudio("1", Buffer.alloc(4));
|
||||
// ...and comes back on the same port.
|
||||
w = await fakeWorker(port);
|
||||
cleanups.push(() => void w.wss.close());
|
||||
await until(() => readies === 2);
|
||||
expect(w.received[0]).toMatchObject({ type: "hello", token: TOKEN });
|
||||
expect(w.received.filter(Buffer.isBuffer)).toEqual([]);
|
||||
});
|
||||
|
||||
it("settings changed with configure() are sent now and in every later hello", async () => {
|
||||
let w = await fakeWorker();
|
||||
const port = w.port;
|
||||
const c = client(w.url, { stt: { language: "ko" } });
|
||||
c.start();
|
||||
await c.waitReady(2000);
|
||||
c.configure({ partials: false });
|
||||
await until(() => w.received.length >= 2);
|
||||
expect(w.received[1]).toEqual({ type: "config", partials: false });
|
||||
for (const s of w.sockets) s.terminate();
|
||||
await new Promise<void>((r) => w.wss.close(() => r()));
|
||||
await until(() => !c.isReady);
|
||||
w = await fakeWorker(port);
|
||||
cleanups.push(() => void w.wss.close());
|
||||
await until(() => c.isReady);
|
||||
expect(w.received[0]).toMatchObject({ stt: { language: "ko", partials: false } });
|
||||
});
|
||||
|
||||
it("tts while disconnected fails at once instead of waiting forever", async () => {
|
||||
const c = client("ws://127.0.0.1:1");
|
||||
const done: TtsDone[] = [];
|
||||
c.on("tts_done", (d) => done.push(d));
|
||||
expect(c.tts("r1", "x")).toBe(false);
|
||||
await until(() => done.length === 1);
|
||||
expect(done[0]).toMatchObject({ id: "r1", failed: true });
|
||||
});
|
||||
});
|
||||
85
app/test/workerProcess.test.ts
Normal file
85
app/test/workerProcess.test.ts
Normal file
@@ -0,0 +1,85 @@
|
||||
import { mkdtempSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { once } from "node:events";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { VoiceWorkerProcess } from "../src/core/workerProcess.js";
|
||||
|
||||
// A stand-in `voice_worker` package run by a real Python, so the spawn line is the one the app uses.
|
||||
const PYTHON = process.env.JARVIS_TEST_PYTHON ?? (process.platform === "win32" ? "python" : "python3");
|
||||
const FAKE_MAIN = `
|
||||
import json, os, sys, time
|
||||
out = os.environ["FAKE_OUT"]
|
||||
runs = len(os.listdir(out))
|
||||
with open(os.path.join(out, f"run{runs}.json"), "w") as f:
|
||||
json.dump({"argv": sys.argv[1:], "token": os.environ.get("JARVIS_VOICE_TOKEN")}, f)
|
||||
if os.environ.get("FAKE_CRASH_FIRST") == "1" and runs == 0:
|
||||
sys.exit(3)
|
||||
print("loading models...", flush=True)
|
||||
print("READY 127.0.0.1:" + sys.argv[sys.argv.index("--port") + 1], flush=True)
|
||||
time.sleep(60)
|
||||
`;
|
||||
|
||||
function fakeWorkerDir() {
|
||||
const root = mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "jarvis-fake-worker-"));
|
||||
mkdirSync(join(root, "voice_worker"));
|
||||
writeFileSync(join(root, "voice_worker", "__init__.py"), "");
|
||||
writeFileSync(join(root, "voice_worker", "__main__.py"), FAKE_MAIN);
|
||||
const out = join(root, "out");
|
||||
mkdirSync(out);
|
||||
return { root, out, run: (n: number) => JSON.parse(readFileSync(join(out, `run${n}.json`), "utf8")) };
|
||||
}
|
||||
|
||||
const procs: VoiceWorkerProcess[] = [];
|
||||
afterEach(async () => {
|
||||
for (const p of procs.splice(0)) await p.stop(1000);
|
||||
});
|
||||
|
||||
describe("VoiceWorkerProcess", () => {
|
||||
it("runs python -m voice_worker on loopback with the token only in the environment", async () => {
|
||||
const d = fakeWorkerDir();
|
||||
const p = new VoiceWorkerProcess({ python: PYTHON, cwd: d.root, args: ["--melo-python", "/x/python"],
|
||||
env: { FAKE_OUT: d.out } });
|
||||
procs.push(p);
|
||||
const ready = once(p, "ready");
|
||||
await p.start();
|
||||
await ready;
|
||||
const run = d.run(0);
|
||||
const port = new URL(p.url).port;
|
||||
expect(run.argv).toEqual(["--host", "127.0.0.1", "--port", port, "--melo-python", "/x/python"]);
|
||||
expect(run.token).toBe(p.token);
|
||||
expect(p.token.length).toBeGreaterThanOrEqual(40);
|
||||
expect(run.argv.join(" ")).not.toContain(p.token);
|
||||
expect(new URL(p.url).hostname).toBe("127.0.0.1");
|
||||
});
|
||||
|
||||
it("restarts a worker that dies, on the same port with the same token", async () => {
|
||||
const d = fakeWorkerDir();
|
||||
const p = new VoiceWorkerProcess({ python: PYTHON, cwd: d.root, args: [], restartDelayMs: 50,
|
||||
env: { FAKE_OUT: d.out, FAKE_CRASH_FIRST: "1" } });
|
||||
procs.push(p);
|
||||
const exited = once(p, "exit");
|
||||
const ready = once(p, "ready");
|
||||
await p.start();
|
||||
expect((await exited)[0]).toMatchObject({ code: 3 });
|
||||
await ready;
|
||||
expect(p.restarts).toBe(1);
|
||||
expect(d.run(1).argv).toEqual(d.run(0).argv);
|
||||
expect(d.run(1).token).toBe(d.run(0).token);
|
||||
});
|
||||
|
||||
it("stop ends the worker for good", async () => {
|
||||
const d = fakeWorkerDir();
|
||||
const p = new VoiceWorkerProcess({ python: PYTHON, cwd: d.root, args: [], restartDelayMs: 20,
|
||||
env: { FAKE_OUT: d.out } });
|
||||
const ready = once(p, "ready");
|
||||
await p.start();
|
||||
await ready;
|
||||
let exits = 0;
|
||||
p.on("exit", () => exits++);
|
||||
await p.stop(2000);
|
||||
await new Promise((r) => setTimeout(r, 200));
|
||||
expect(exits).toBe(1);
|
||||
expect(p.restarts).toBe(0);
|
||||
});
|
||||
});
|
||||
89
app/test/workerProtocol.test.ts
Normal file
89
app/test/workerProtocol.test.ts
Normal file
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* The app's worker client against the real Python voice worker server (with fake engines), so both
|
||||
* sides of the protocol are checked together. Needs a Python with the worker's requirements:
|
||||
* JARVIS_VOICE_PYTHON=<venv python> npm test
|
||||
*/
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { createInterface } from "node:readline";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { Playback } from "../src/core/playback.js";
|
||||
import { VoiceWorkerClient, type Final, type TtsDone } from "../src/core/workerClient.js";
|
||||
import { freePort } from "../src/core/workerProcess.js";
|
||||
|
||||
const PYTHON = process.env.JARVIS_VOICE_PYTHON;
|
||||
const SRC = fileURLToPath(new URL("../../src", import.meta.url));
|
||||
const FIXTURE = fileURLToPath(new URL("./fixtures/fake_engine_worker.py", import.meta.url));
|
||||
const TOKEN = "protocol-test-token";
|
||||
const FMT = { rate: 48000, channels: 2 } as const;
|
||||
|
||||
/** 20 ms of a 48 kHz stereo tone whose amplitude and pitch (x1000 Hz) are `level`. */
|
||||
function frame(level: number): Buffer {
|
||||
const n = 960;
|
||||
const buf = Buffer.alloc(n * 4);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const v = Math.round(level * Math.sin((2 * Math.PI * level * 1000 * i) / 48000) * 32767);
|
||||
buf.writeInt16LE(v, i * 4);
|
||||
buf.writeInt16LE(v, i * 4 + 2);
|
||||
}
|
||||
return buf;
|
||||
}
|
||||
|
||||
describe.skipIf(!PYTHON)("worker protocol against the Python server", () => {
|
||||
let child: ChildProcess;
|
||||
let client: VoiceWorkerClient;
|
||||
|
||||
beforeAll(async () => {
|
||||
const port = await freePort();
|
||||
child = spawn(PYTHON!, [FIXTURE, String(port)], {
|
||||
env: { ...process.env, PYTHONPATH: SRC, JARVIS_VOICE_TOKEN: TOKEN },
|
||||
stdio: ["ignore", "pipe", "inherit"],
|
||||
});
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
child.once("exit", (c) => reject(new Error(`fake worker exited ${c}`)));
|
||||
createInterface({ input: child.stdout! }).on("line", (l) => l.startsWith("READY") && resolve());
|
||||
});
|
||||
client = new VoiceWorkerClient({ url: `ws://127.0.0.1:${port}`, token: TOKEN, audio: FMT, tts: FMT,
|
||||
stt: { partials: false, hotwords: ["자비스"] } });
|
||||
client.start();
|
||||
const ok = await client.waitReady(5000);
|
||||
expect(ok).toMatchObject({ stt_ready: true, tts_ready: true });
|
||||
expect(ok.stt).toMatchObject({ partials: false, hotwords: ["자비스"] });
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
client?.close();
|
||||
child?.kill();
|
||||
});
|
||||
|
||||
it("Discord-format audio from two speakers comes back as each speaker's final", async () => {
|
||||
const finals: Final[] = [];
|
||||
client.on("final", (f) => finals.push(f));
|
||||
for (let i = 0; i < 30; i++) {
|
||||
client.sendAudio("111111111111111111", frame(0.9));
|
||||
client.sendAudio("222222222222222222", frame(0.6));
|
||||
}
|
||||
client.speakingEnd("111111111111111111");
|
||||
// The second speaker leaves mid-utterance: the final still comes.
|
||||
client.speakerLeave("222222222222222222");
|
||||
const end = Date.now() + 5000;
|
||||
while (finals.length < 2 && Date.now() < end) await new Promise((r) => setTimeout(r, 20));
|
||||
const bySpeaker = Object.fromEntries(finals.map((f) => [f.speaker, f]));
|
||||
expect(bySpeaker["111111111111111111"]).toMatchObject({ text: "level 0.9", end_reason: "speaking_end", utt: 1 });
|
||||
expect(bySpeaker["222222222222222222"]).toMatchObject({ text: "level 0.6", end_reason: "speaking_end", utt: 1 });
|
||||
});
|
||||
|
||||
it("TTS sentences play in order through Playback and the request ends cleanly", async () => {
|
||||
const played: Buffer[] = [];
|
||||
const playback = new Playback(client, { play: (pcm) => { played.push(pcm); setTimeout(() => playback.clipEnded(), 5); }, stop: () => {} });
|
||||
client.on("tts_audio", (f) => playback.audio(f.id, f.seq, f.pcm));
|
||||
client.on("tts_done", (d: TtsDone) => playback.done(d));
|
||||
const ended = new Promise((resolve) => playback.once("end", resolve));
|
||||
const text = "첫 번째 문장입니다. 두 번째 문장은 조금 더 깁니다.";
|
||||
playback.say(text);
|
||||
expect(await ended).toMatchObject({ played: 2, cancelled: false, failed: false });
|
||||
// 10 samples per character at 48 kHz, mono duplicated to stereo, 2 bytes each.
|
||||
const sentences = ["첫 번째 문장입니다.", "두 번째 문장은 조금 더 깁니다."];
|
||||
expect(played.map((p) => p.length)).toEqual(sentences.map((s) => s.length * 10 * 2 * 2));
|
||||
});
|
||||
});
|
||||
17
app/tsconfig.json
Normal file
17
app/tsconfig.json
Normal file
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2023",
|
||||
"lib": ["ES2023"],
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"sourceMap": true,
|
||||
"types": ["node"]
|
||||
},
|
||||
"include": ["src"]
|
||||
}
|
||||
10
app/vitest.config.ts
Normal file
10
app/vitest.config.ts
Normal file
@@ -0,0 +1,10 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ["test/**/*.test.ts"],
|
||||
pool: "forks",
|
||||
poolOptions: { forks: { maxForks: 2, minForks: 1 } },
|
||||
testTimeout: 15_000,
|
||||
},
|
||||
});
|
||||
@@ -6,7 +6,7 @@
|
||||
|---|---|---|
|
||||
| M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. 가상 모니터 확인을 뺀 모든 항목 측정 완료(2026-11-10). 가상 모니터 확인은 M6에서 합니다. 2026-11-10에 추가한 `desktop` 단계(agent-desktop)는 측정을 마쳤고, 자동 성능 조절용 `gamemode` 단계는 세 번 쟀고, `gamemode_stt`(LLM과 STT를 함께 켬, STT GPU·CPU)와 `gamemode_live`(실제 게임을 켬) 단계도 측정을 마쳤습니다(2026-11-10). 결과로 게임 중 최소 배치는 전문가 2GB로 정했고, 평소 배치에 STT 자리를 남기는 것과 게임 중 LLM이 답하는 동안 부분 전사 간격을 늘리는 것(게임이 없을 때는 그대로)도 정했습니다(사용자 결정, 2026-11-10). `gamemode_tune` 단계도 측정을 마쳤고(2026-11-11), 게임 중 LLM이 답하는 동안에는 부분 전사를 멈추기로 정했습니다. GPU 95% 제한은 측정에서 사용률을 낮추지 못해 기본값을 끔으로 바꿨고(사용자 결정, 2026-11-11), `video_drops` 단계(Jarvis를 돌리는 동안 유튜브 영상의 끊긴 프레임, 직접 실행)도 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 영상이 끊기지 않았고 제한이 끊김을 줄이지 못해 설정을 뺐습니다. 대신 게임 화면의 프레임을 직접 재는 `game_frames` 단계(게임을 켠 채로 Jarvis를 끈 때·쉴 때·바쁠 때와 우선순위를 낮춘 양보 상태, 직접 실행, 관리자 권한)를 넣었고 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 게임의 평균 fps와 1% low가 5% 넘게 떨어지지 않아 양보는 넣지 않기로 했습니다. 남은 M0 측정은 가상 모니터 확인(M6)입니다. 측정 키트: [scripts/m0/](../../scripts/m0/README.md) |
|
||||
| M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과. 구현과 테스트 완료(2026-11-11): `src/voice_worker/`, 스펙은 [voice_worker.spec.md](../../src/voice_worker/voice_worker.spec.md). 이 서버의 RTX 5050에서 10명이 겹쳐 말하는 실시간 입력(48kHz 스테레오, M0 한국어 샘플 8개를 10명에게 나눠 줌, 절반은 디스코드 speaking 종료, 절반은 무음으로 끝남)을 넣어 10명 모두 최종 전사를 받았고 문자 오류율은 평균 4%, 최대 19%("일곱 시"를 "7시"로 적은 것), 말 끝 판정부터 최종 전사 전송까지 중앙값 0.64초, 최대 1.56초였습니다. TTS는 첫 문장 소리가 0.2초에 나왔고, 그 소리를 다시 넣으면 같은 문장으로 받아 적혔습니다. 샘플이 MeloTTS 합성 음성이라 실제 사람 녹음과 게임PC 수치는 아직 없습니다 |
|
||||
| M2 | 디스코드 음성 코어 이전 | v1 봇 코드를 Electron 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기) |
|
||||
| M2 | 디스코드 음성 코어 이전 | v1 봇 코드를 Electron 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기). 구현과 테스트 완료(2026-11-11): `app/`(Electron 45 + TypeScript), 스펙은 [discord_core.spec.md](../../app/src/core/discord_core.spec.md). 앱이 음성 워커를 띄우고(실행마다 새 토큰, 죽으면 다시 시작) 연결이 끊기면 다시 붙습니다. 음성 채널에 들어갈 때마다 전사 고지를 채널 채팅에 올립니다. 실제 디스코드에서 봇 두 개로 확인(Electron 런타임, 이 서버의 RTX 5050): 다른 봇이 채널에 튼 녹음 문장 5개를 모두 받아 적었고(문자 오류율 4개 0%, "일곱 시"를 "7시"로 적은 1개 12.5%), 말이 끝난 뒤 최종 전사까지 0.66~0.83초였습니다. 자비스가 말한 문장을 상대 봇이 받아 다시 전사하니 오류율 3.4%였고, 정지 명령 뒤 0.02초 만에 소리가 끊겼습니다. 채팅 채널은 새 글 받기, 최근 글 읽기, 보내기(2,000자 넘으면 나눔, 아무도 호출하지 않음)가 됩니다. 사람이 직접 말한 소리와 게임PC(Windows)에서는 아직 확인하지 않았고, 음성으로 "자비스, 멈춰"는 호출 판정이 필요해 M4에서 합니다 |
|
||||
| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과 |
|
||||
| M4 | 왕복 완성과 지연 튜닝 | 호출어·호출 방식·지정한 사람 설정, 동시 대화 응답 정책 4가지(기본 "처음 대화만")와 정책별 끼어들기 규칙. 말 끝부터 첫 소리까지 실측 1.5초 이내, 10명 채널 부하 테스트 |
|
||||
| M5 | 관리 앱 기본 | 첫 실행 마법사, 데이터 폴더, 런타임·모델 자동 설치, 대시보드, 봇/채널·호출/채팅 채널(디스코드 채널 이름·주제 반영 포함)/대화 기억/모델 설정, 로그 화면(보기·복사·다운로드) |
|
||||
|
||||
Reference in New Issue
Block a user