joinChannel reused a same-guild connection and re-subscribed a new player and
receiver each time, stacking duplicate speaking listeners (→ duplicate voice
turns) and error handlers. Now it no-ops if already in the target channel and
otherwise leaves the current connection first, so channel switches are clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a dashboard<->bot control plane (bot pushes state + polls commands, keeping
the bot's single outbound-HTTP direction):
- New bot_control.BotControl + endpoints: GET /api/bot/state, /api/bot/commands;
POST /api/bot/report, /api/bot/select.
- Dashboard header bar: bot identity/connection, server dropdown (top "없음"),
voice-channel dropdown (top "없음"), and live participant list.
- Turns record who spoke (Turn.speaker, via X-User-Name on the voice-turn POST).
- dave/bot.mjs: reports identity/guilds/voice-channels/members, polls join/leave
commands and joins dynamically, and sends the speaker's display name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Bottom-docked collapsible terminal log panel (open/close), retains the event
log from voice-server start (event tail 200→2000).
- Log search box + level filter (전체/오류/경고/정보); per-line 삭제/수정 and
전체 삭제, backed by new monitor event ids and /api/logs/{clear,delete,edit}.
- Turns now show 들음 / 생각 / 답변 three rows; 생각 surfaces the emotion-tone
plan the bot chose (and the [잡음] decision), via a new Turn.thought field.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a persisted, runtime-editable system prompt. The brain reads the persona
on every turn (prompt_store.get_persona), so a dashboard edit applies to the
next reply with no restart; blank clears the override back to the built-in
PERSONA. New endpoints GET/POST /api/prompt, and a reusable modal popup
(뒤로가기 + 수정/저장) that later white/blacklist features will share.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The dashboard voice-turn path ran _speech_text() before MeloTTS.synth, which
rewrote a leading "[힘차게] 안녕!" into "힘차게, 안녕!" — reading the first
emotion aloud and destroying the tag before the TTS emotion parser could use it.
Make _speech_text() a pass-through so every emotion tag (including the first)
reaches synth intact and shapes pitch/speed instead of being spoken. Adds a
regression test covering the leading-tag case.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Emotion tags now steer delivery rather than being read aloud. parse_segments()
splits a reply on [감정] tags: a recognised emotion word switches the pitch and
speed of the text that follows (and is dropped), while a non-emotion bracket
(e.g. [1번]) keeps its inner words as spoken content. Emotions can change
mid-reply, so a single turn is synthesised as several pitch-shifted segments and
concatenated in the melo worker (librosa pitch_shift, warmed at startup).
The emotion vocabulary is grounded in Azure Neural TTS speaking styles plus
Ekman's basic emotions, with Korean synonyms. The brain persona is updated to
emit inline tags from that set.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Brain persona now prefixes every reply with one bracketed emotion tag
(e.g. [반가움], [궁금]) and keeps replies to one or two short sentences.
- Empty/unrecognised audio (silence/noise) is reported as reply "[잡음]" with
no TTS playback instead of an empty reply.
- TTS speaks the bracketed emotion too: "[힘차게] 안녕!" is synthesised as
"힘차게, 안녕!" via a leading-tag -> spoken-word transform.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Log voice connection errors instead of letting them surface silently.
- Collapse bursty repeated receive-stream errors (DAVE E2EE group-transition
decrypt failures) into one line + a suppressed-count summary, so a member
joining/leaving no longer floods the log.
Deploy: voice-server now runs as the wsai-voice.service user unit (STT+Claude
Haiku brain+TTS on GPU); the bot container reaches it via
host.docker.internal:8787.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude replies with markdown/backticks by default; MeloTTS's Korean text
normaliser has no entry for '`' and dies with KeyError: '`', so any reply
mentioning a command/code block crashed the whole voice turn (500 on
/api/voice-turn). Fix at the shared synth() choke point with
normalize_for_speech(), which flattens code fences/inline code/links/markdown
and guarantees no backtick reaches the worker — covering both the dashboard
voice turn and the Discord speak() bridge. Also add a PERSONA line asking the
model to avoid markdown (belt-and-suspenders; the code strip is the real fix).
errors_total never moved for turn-level failures: it was only bumped by
log("error") events, and the dashboard voice path calls turn.finish(error=...)
without logging. Emit one error-level log event from Turn.finish() when a turn
ends in error, so both the server counter and the browser SSE mirror stay
consistent, guarded to count at most once. Drop the now-redundant pipeline
log("error") to avoid double counting and remove the dead _publish stub.
Verified: raw backtick -> worker KeyError '`' reproduced; after fix real
MeloTTS synth of a backtick+fenced reply succeeds; /api/voice-turn returns 200
with a wav body on a backtick reply and errors_total stays 0, and an induced
synth failure returns 500 with errors_total incrementing to exactly 1. Full
suite 18 passed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A single Claude 529 Overloaded dropped the voice turn straight to the apology
fallback. The anthropic SDK retries >=500/429 but only twice by default, which
a busy window can outlast. Raise max_retries (WSAI_BRAIN_MAX_RETRIES, default 4)
so transient overloads recover silently, and give overloads their own spoken
fallback ("서버가 붐벼서...") distinct from generic failures. Kept modest so a
sustained outage still fails fast instead of leaving the bot silent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The voice loop echoed the recognised text. Wire the real brain: the Discord
voice-turn now runs STT -> ClaudeBrain.respond (with rolling conversation
history) -> TTS, so the bot actually thinks and answers. --voice-server builds
the brain by default (WSAI_BRAIN=claude, WSAI_BRAIN_MODEL overridable) and
gracefully falls back to echo if anthropic/Claude auth is unavailable. A brain
error speaks a short apology instead of killing the loop.
Verified end-to-end: an utterance wav returns X-Heard plus a distinct Claude
X-Reply and a synthesised reply wav on device=cuda. 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Raise the voice-Ready ceiling 20s->40s: the DAVE/MLS handshake cycles
signalling<->connecting and can take ~25s, so 20s spuriously failed the join.
- Handle AudioReceiveStream 'error' (e.g. a DAVE decrypt/UDP GenericFailure on
one packet): log and free the speaker slot instead of letting the unhandled
'error' event crash the whole bot process.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bot (dave/bot.mjs) previously only joined the channel and counted audio
frames — it never fed STT or spoke back. Wire the real loop:
- Node bot: buffer each speaker's Opus->PCM utterance until AfterSilence,
wrap as WAV, POST to the Python voice-turn endpoint, then play the returned
reply wav into the channel via an AudioPlayer (ffmpeg->Opus). Skips its own
audio, dedupes overlapping subscriptions, and ignores sub-0.35s noise.
- Python: new `python -m wsai --voice-server` serves /api/voice-turn — decode
the uploaded utterance, GPU faster-whisper STT, produce a reply (echo of what
was heard for now), GPU MeloTTS synth, return the reply wav (recognised/reply
text ride along as X-Heard/X-Reply headers). Both engines pre-warmed; turns
show in the dashboard feed. MeloTTS.synth() extracted for direct wav reuse.
Echo mode verifies listening+speaking+GPU recognition entirely in Discord; the
Claude brain is the next slice. Verified the endpoint round-trip: utterance wav
-> correct Korean X-Heard/X-Reply + a WAVE reply on device=cuda. 12 tests pass,
node --check clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The status page was view-only, so there was no way to actually verify Korean
recognition end-to-end. Add a live test: record from the mic (localhost/https)
or upload an audio file (works over LAN http, where browsers block getUserMedia),
POST it to a new /api/stt endpoint that ffmpeg-normalises the blob to 16 kHz
mono and runs the real GPU faster-whisper, then shows the recognised text +
latency + device. Results also land in the live turn feed.
The dashboard now optionally holds a WhisperSTT and drives it from a private
asyncio loop thread. New `python -m wsai --stt-test` serves the page with STT
enabled and pre-warms the GPU worker so the first recognition is instant.
WhisperSTT.resolved_device is exposed for the UI.
Verified: wav and browser-style webm/opus uploads both return the correct
Korean text on device=cuda in ~240-280ms. 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The first CUDA inference pays a large lazy cost (kernel autotune/cudnn) —
~10s for a cold TTS synth — which would blow the voice loop's ~1s budget on
the very first reply. Each worker now runs one dummy inference (TTS: a short
phrase; STT: 1s of silence) after model load and before emitting "ready", so
"ready" means "hot". Warmup failures are logged and never block startup.
Verified: first real call after startup is now TTS ~238ms / STT ~189ms
(was ~11s cold for TTS). 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both voice backends defaulted to CPU. Fix the "CUDA unavailable" gaps so
everything that benefits from the RTX 5050 uses it:
- MeloTTS venv had CPU-only torch (2.12.0+cpu) -> installed Blackwell-capable
torch/torchaudio 2.11.0+cu128 (sm_120 verified with a real GPU matmul).
- faster-whisper CUDA loaded but transcribe() died with "libcublas.so.12 not
found": installed nvidia-cublas-cu12 + nvidia-cudnn-cu12 into the whisper
venv and inject those nvidia/*/lib dirs into the worker's LD_LIBRARY_PATH at
spawn (the loader only honours it at exec).
- WSAI_WHISPER_DEVICE / WSAI_MELO_DEVICE now default to "auto": pick CUDA when
present, else CPU, and each worker falls back to CPU if a CUDA load fails so
the voice loop never dies on a GPU-less host.
Verified end-to-end through the real backend classes: both workers report
"ready on cuda"; steady-state STT ~170ms (was ~1350ms CPU), TTS ~4s first call
vs ~23s CPU. All 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Step 3 (귀): add WhisperSTT + whisper_worker, a warm out-of-venv worker
mirroring the MeloTTS shape (whisper312 venv, small/int8 on CPU). transcribe()
closes the voice round trip (MeloTTS wav -> whisper text); utterances() turns an
injected audio_source into Utterances (Discord voice feed pending). Wired into
factory as WSAI_STT=whisper.
Also address the arbiter's TTS follow-ups:
- melo worker error handling: capture stderr (drained in a bounded background
task so the pipe can't fill), surface the real failure cause, and defend
against an empty/invalid ready line instead of dying on JSONDecodeError.
- pipeline pre-warm: load slow backends (warmup()) at startup so the first
utterance is answered warm; a warmup failure is logged, not fatal.
Verified: real TTS->STT round trip recovers the sentence near-perfectly;
warm transcribe ~1.2s (CPU). 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a MeloTTS backend that runs the model in its own melo311 interpreter
as a long-lived worker (melo_worker.py), loaded once and fed synthesis
requests over a stdin/stdout JSON protocol. fd1 is split from fd2 in the
worker so MeloTTS's stdout progress chatter can't corrupt the protocol.
Each speak() writes a wav and hands the path to a pluggable sink (the
Discord voice step will swap in "play into the call"). factory wires
tts=melo; pipeline.aclose now also tears down the tts worker.
Verified (CPU): model load ~7.9s once, then a short reply synthesizes in
~0.86s (within the ~1s budget); wav is valid 44.1kHz PCM. GPU (cuda) is
selectable via WSAI_MELO_DEVICE for lower latency, pending GPU approval.
7 smoke tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wires the deployment's Claude Max login (OAuth token from
$CLAUDE_CREDENTIALS_PATH) into ClaudeBrain/ClaudeVision. OAuth tokens
authenticate as Bearer (auth_token=), not x-api-key, and only answer
when the first system block is the Claude Code identity string, so the
real persona moves to a second system block. Token is re-read per
request so a host-side refresh is picked up without a restart. Falls
back to ANTHROPIC_API_KEY when set.
Verified: WSAI_BRAIN=claude returns a real Korean reply through the
factory; 7 smoke tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
--dashboard defaulted to looping mock STT forever, so the status page
piled up thousands of fake "conversations" (all mock, 0ms, same reply)
that looked like real traffic. Now it plays 3 sample utterances then
idles; a loud 데모 모드 banner states the turns are mock samples, not
real STT/Brain/TTS. Continuous demo moved behind --dashboard-loop-demo.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a stdlib-only observability site so you can open a browser and watch,
step by step: whether it is listening, what it heard, what the brain thought
and answered, how long each stage took, and whether anything errored.
- wsai/monitor.py: thread-safe telemetry hub (per-turn timed steps, status
header, error log) with a pub/sub for live push.
- wsai/dashboard.py: stdlib http.server serving a self-contained page plus an
SSE (/events) live stream; /api/state snapshot fallback.
- Pipeline emits step-by-step turn telemetry (화면 맥락 → 두뇌 → 응답) and
listening/running status; optional monitor, so existing paths are untouched.
- `python -m wsai --dashboard` starts the site (0.0.0.0:8787, WSAI_DASHBOARD_PORT)
and loops the mock voice demo so there is always live activity to watch.
- Tests cover turn recording, per-step timing, error marking, and live push.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The `join` entrypoint command now execs `node dave/bot.mjs` and forwards args;
Dockerfile comments/env notes updated from DISCORD_SELFBOT_TOKEN to the official
bot path. Rebuilt watch_sceen_ai:test on .9 and verified in-container:
* smoke -> 4 passed
* voice -> eyes-free mock loop runs
* join -> LIVE bot join (테스트봇#9029) to guild 사지방 / channel 일반, Ready, clean leave
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the user-token selfbot voice path with an official Discord bot using
discord.js 14 + @discordjs/voice 0.19. The bot logs in with the stored testbot
token, joins the target voice channel, passes the DAVE/MLS E2EE handshake, and
receives per-user Opus audio via VoiceReceiver (the STT input path). ToS-safe.
Live-verified: bot joined guild "사지방" / channel "일반" and reached Ready.
Selfbot (gate.mjs/join.mjs) kept only for the deferred screenshare-video track,
which official bots cannot receive. Docs updated (README/PLAN); M1 done on bot path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pipeline.run() used asyncio.gather, so if one loop raised, the failing
coroutine propagated while the sibling loops kept running detached; aclose()
in the finally then closed a source/stt out from under a still-live loop.
Switch to asyncio.TaskGroup so a failing loop cancels+awaits the siblings
before teardown. Add a regression test asserting an error in the conversation
loop cancels the perception loop and still closes every source.
The RUN_MS auto-leave was only armed inside announceReady(), which requires a
fully successful join (DAVE/MLS op29/op30 -> mlsReady). When the E2EE handshake
stalls after op26 key_package, announceReady never fires, so the "time-boxed"
selfbot ran unbounded in a live channel. Arm RUN_MS at process startup instead,
independent of handshake state, and make leaveAndExit idempotent so the ceiling,
ready timer, and signal handlers can't double-fire.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bundles the M1 milestone: mock voice pipeline (wsai) + dave/ selfbot voice
joiner. Entrypoint dispatches smoke/voice/mock/gpu/join/shell. .env mounted at
runtime (not baked). Built GPU-capable via NVIDIA CDI so later STT/TTS drop in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
join.mjs joins the target voice channel over the proven DAVE handshake and stays
connected, mapping SPEAKING->ssrc and tallying incoming RTP (foundation for M2
audio decrypt). README records GPU=on, shared OAuth brain, natural-but-<=1s TTS,
Discord-voice STT input, and the M1..M6 milestones.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Make source/vision optional so the conversation loop runs with no screen
capture. Add Settings.voice() preset and `python -m wsai --voice` demo.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fail-fast checkpoint the arbiter mandated before committing to option A
(protocol-level selfbot stream receive). dave/gate.mjs proves, live against
Discord, that a user token can pass the voice DAVE/MLS handshake:
- voice GW v8 IDENTIFY with max_dave_protocol_version=1 -> NO close 4017
- dave_protocol_version=1 negotiated (E2EE active on the channel)
- @snazzah/davey drives full MLS membership: op25 external_sender -> op26
key_package -> op27 proposals -> op28 commit_welcome -> op29 announce_commit
-> MLS session ready=true, stable 5s, voicePrivacyCode derived
Confirms option A is viable: the selfbot can join the E2EE group as a full
member, which is the prerequisite for receiving+decrypting the video RTP.
PLAN.md updated with gate result, exact binary framing, and next A steps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
User rejected browser/screenshot capture: they want to actually receive the
real-time screen-share stream like a real client. Research confirms this needs a
selfbot (user token) via discord.js-selfbot-v13 (werift-rtp) exposing incoming
video RTP (receiverData/hasVideo, VoiceReceiver, StreamConnection); decode via
libsodium decrypt + depayload + ffmpeg. Records ToS/ban risk and that no
turnkey receive->frames library exists. Xvfb browser PoC demoted to fallback.
Decision #1: watch the real Discord screen share by running a real web client
under a virtual display (Xvfb) and capturing rendered frames — not bot/selfbot
protocol receive. PLAN.md records all locked decisions and the step list.
PoC (poc/capture_xvfb.py + test_page.html) launches system Chrome non-headless
under Xvfb via Playwright and captures 6 changing, non-blank frames -> de-risks
the display+capture chain on the .9 host.
Modular async pipeline: FrameSource->Vision->context and STT/text->Brain->TTS.
All stages are Protocols; mock backends run end-to-end with no deps/keys.
Real backends included: mss screen capture, Claude vision+brain (guarded imports).
