Both voice backends defaulted to CPU. Fix the "CUDA unavailable" gaps so
everything that benefits from the RTX 5050 uses it:
- MeloTTS venv had CPU-only torch (2.12.0+cpu) -> installed Blackwell-capable
torch/torchaudio 2.11.0+cu128 (sm_120 verified with a real GPU matmul).
- faster-whisper CUDA loaded but transcribe() died with "libcublas.so.12 not
found": installed nvidia-cublas-cu12 + nvidia-cudnn-cu12 into the whisper
venv and inject those nvidia/*/lib dirs into the worker's LD_LIBRARY_PATH at
spawn (the loader only honours it at exec).
- WSAI_WHISPER_DEVICE / WSAI_MELO_DEVICE now default to "auto": pick CUDA when
present, else CPU, and each worker falls back to CPU if a CUDA load fails so
the voice loop never dies on a GPU-less host.
Verified end-to-end through the real backend classes: both workers report
"ready on cuda"; steady-state STT ~170ms (was ~1350ms CPU), TTS ~4s first call
vs ~23s CPU. All 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Step 3 (귀): add WhisperSTT + whisper_worker, a warm out-of-venv worker
mirroring the MeloTTS shape (whisper312 venv, small/int8 on CPU). transcribe()
closes the voice round trip (MeloTTS wav -> whisper text); utterances() turns an
injected audio_source into Utterances (Discord voice feed pending). Wired into
factory as WSAI_STT=whisper.
Also address the arbiter's TTS follow-ups:
- melo worker error handling: capture stderr (drained in a bounded background
task so the pipe can't fill), surface the real failure cause, and defend
against an empty/invalid ready line instead of dying on JSONDecodeError.
- pipeline pre-warm: load slow backends (warmup()) at startup so the first
utterance is answered warm; a warmup failure is logged, not fatal.
Verified: real TTS->STT round trip recovers the sentence near-perfectly;
warm transcribe ~1.2s (CPU). 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a MeloTTS backend that runs the model in its own melo311 interpreter
as a long-lived worker (melo_worker.py), loaded once and fed synthesis
requests over a stdin/stdout JSON protocol. fd1 is split from fd2 in the
worker so MeloTTS's stdout progress chatter can't corrupt the protocol.
Each speak() writes a wav and hands the path to a pluggable sink (the
Discord voice step will swap in "play into the call"). factory wires
tts=melo; pipeline.aclose now also tears down the tts worker.
Verified (CPU): model load ~7.9s once, then a short reply synthesizes in
~0.86s (within the ~1s budget); wav is valid 44.1kHz PCM. GPU (cuda) is
selectable via WSAI_MELO_DEVICE for lower latency, pending GPU approval.
7 smoke tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wires the deployment's Claude Max login (OAuth token from
$CLAUDE_CREDENTIALS_PATH) into ClaudeBrain/ClaudeVision. OAuth tokens
authenticate as Bearer (auth_token=), not x-api-key, and only answer
when the first system block is the Claude Code identity string, so the
real persona moves to a second system block. Token is re-read per
request so a host-side refresh is picked up without a restart. Falls
back to ANTHROPIC_API_KEY when set.
Verified: WSAI_BRAIN=claude returns a real Korean reply through the
factory; 7 smoke tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
--dashboard defaulted to looping mock STT forever, so the status page
piled up thousands of fake "conversations" (all mock, 0ms, same reply)
that looked like real traffic. Now it plays 3 sample utterances then
idles; a loud 데모 모드 banner states the turns are mock samples, not
real STT/Brain/TTS. Continuous demo moved behind --dashboard-loop-demo.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a stdlib-only observability site so you can open a browser and watch,
step by step: whether it is listening, what it heard, what the brain thought
and answered, how long each stage took, and whether anything errored.
- wsai/monitor.py: thread-safe telemetry hub (per-turn timed steps, status
header, error log) with a pub/sub for live push.
- wsai/dashboard.py: stdlib http.server serving a self-contained page plus an
SSE (/events) live stream; /api/state snapshot fallback.
- Pipeline emits step-by-step turn telemetry (화면 맥락 → 두뇌 → 응답) and
listening/running status; optional monitor, so existing paths are untouched.
- `python -m wsai --dashboard` starts the site (0.0.0.0:8787, WSAI_DASHBOARD_PORT)
and loops the mock voice demo so there is always live activity to watch.
- Tests cover turn recording, per-step timing, error marking, and live push.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The `join` entrypoint command now execs `node dave/bot.mjs` and forwards args;
Dockerfile comments/env notes updated from DISCORD_SELFBOT_TOKEN to the official
bot path. Rebuilt watch_sceen_ai:test on .9 and verified in-container:
* smoke -> 4 passed
* voice -> eyes-free mock loop runs
* join -> LIVE bot join (테스트봇#9029) to guild 사지방 / channel 일반, Ready, clean leave
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the user-token selfbot voice path with an official Discord bot using
discord.js 14 + @discordjs/voice 0.19. The bot logs in with the stored testbot
token, joins the target voice channel, passes the DAVE/MLS E2EE handshake, and
receives per-user Opus audio via VoiceReceiver (the STT input path). ToS-safe.
Live-verified: bot joined guild "사지방" / channel "일반" and reached Ready.
Selfbot (gate.mjs/join.mjs) kept only for the deferred screenshare-video track,
which official bots cannot receive. Docs updated (README/PLAN); M1 done on bot path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pipeline.run() used asyncio.gather, so if one loop raised, the failing
coroutine propagated while the sibling loops kept running detached; aclose()
in the finally then closed a source/stt out from under a still-live loop.
Switch to asyncio.TaskGroup so a failing loop cancels+awaits the siblings
before teardown. Add a regression test asserting an error in the conversation
loop cancels the perception loop and still closes every source.
The RUN_MS auto-leave was only armed inside announceReady(), which requires a
fully successful join (DAVE/MLS op29/op30 -> mlsReady). When the E2EE handshake
stalls after op26 key_package, announceReady never fires, so the "time-boxed"
selfbot ran unbounded in a live channel. Arm RUN_MS at process startup instead,
independent of handshake state, and make leaveAndExit idempotent so the ceiling,
ready timer, and signal handlers can't double-fire.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bundles the M1 milestone: mock voice pipeline (wsai) + dave/ selfbot voice
joiner. Entrypoint dispatches smoke/voice/mock/gpu/join/shell. .env mounted at
runtime (not baked). Built GPU-capable via NVIDIA CDI so later STT/TTS drop in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
join.mjs joins the target voice channel over the proven DAVE handshake and stays
connected, mapping SPEAKING->ssrc and tallying incoming RTP (foundation for M2
audio decrypt). README records GPU=on, shared OAuth brain, natural-but-<=1s TTS,
Discord-voice STT input, and the M1..M6 milestones.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Make source/vision optional so the conversation loop runs with no screen
capture. Add Settings.voice() preset and `python -m wsai --voice` demo.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fail-fast checkpoint the arbiter mandated before committing to option A
(protocol-level selfbot stream receive). dave/gate.mjs proves, live against
Discord, that a user token can pass the voice DAVE/MLS handshake:
- voice GW v8 IDENTIFY with max_dave_protocol_version=1 -> NO close 4017
- dave_protocol_version=1 negotiated (E2EE active on the channel)
- @snazzah/davey drives full MLS membership: op25 external_sender -> op26
key_package -> op27 proposals -> op28 commit_welcome -> op29 announce_commit
-> MLS session ready=true, stable 5s, voicePrivacyCode derived
Confirms option A is viable: the selfbot can join the E2EE group as a full
member, which is the prerequisite for receiving+decrypting the video RTP.
PLAN.md updated with gate result, exact binary framing, and next A steps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
User rejected browser/screenshot capture: they want to actually receive the
real-time screen-share stream like a real client. Research confirms this needs a
selfbot (user token) via discord.js-selfbot-v13 (werift-rtp) exposing incoming
video RTP (receiverData/hasVideo, VoiceReceiver, StreamConnection); decode via
libsodium decrypt + depayload + ffmpeg. Records ToS/ban risk and that no
turnkey receive->frames library exists. Xvfb browser PoC demoted to fallback.
Decision #1: watch the real Discord screen share by running a real web client
under a virtual display (Xvfb) and capturing rendered frames — not bot/selfbot
protocol receive. PLAN.md records all locked decisions and the step list.
PoC (poc/capture_xvfb.py + test_page.html) launches system Chrome non-headless
under Xvfb via Playwright and captures 6 changing, non-blank frames -> de-risks
the display+capture chain on the .9 host.
Modular async pipeline: FrameSource->Vision->context and STT/text->Brain->TTS.
All stages are Protocols; mock backends run end-to-end with no deps/keys.
Real backends included: mss screen capture, Claude vision+brain (guarded imports).
