merge: unify Gitea main README history into code history

The dashboard voice-turn path ran _speech_text() before MeloTTS.synth, which
rewrote a leading "[힘차게] 안녕!" into "힘차게, 안녕!" — reading the first
emotion aloud and destroying the tag before the TTS emotion parser could use it.
Make _speech_text() a pass-through so every emotion tag (including the first)
reaches synth intact and shapes pitch/speed instead of being spoken. Adds a
regression test covering the leading-tag case.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

.dockerignore

@@ -0,0 +1,8 @@
+.venv/
+**/__pycache__/
+*.pyc
+.pytest_cache/
+dave/node_modules/
+poc/frames/
+.git
+.env

.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.pyc
+.pytest_cache/
+.venv/
+*.egg-info/
+poc/frames/
+.env
+node_modules/

Dockerfile

@@ -0,0 +1,46 @@
+# watch_sceen_ai — GPU-ready test image for the .9 host (RTX 5050).
+#
+# Bundles everything the current M1 milestone needs:
+#   * CUDA runtime (so future faster-whisper / TTS can use the GPU)
+#   * Python 3.11 (STT/TTS asset compat per README) for the wsai voice pipeline
+#   * Node 22 + ffmpeg for the dave/ official-bot voice joiner (DAVE/MLS E2EE)
+#
+# STT/TTS/Brain are still mock at this milestone, so nothing GPU-heavy runs yet;
+# the image is built GPU-capable so the next milestones drop straight in.
+FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04
+
+ENV DEBIAN_FRONTEND=noninteractive \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1
+
+# --- system deps: python 3.11, node 22, ffmpeg -----------------------------
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        software-properties-common ca-certificates curl gnupg ffmpeg \
+    && add-apt-repository -y ppa:deadsnakes/ppa \
+    && apt-get update && apt-get install -y --no-install-recommends \
+        python3.11 python3.11-venv python3.11-dev \
+    && curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
+    && apt-get install -y --no-install-recommends nodejs \
+    && ln -sf /usr/bin/python3.11 /usr/local/bin/python \
+    && python -m ensurepip --upgrade \
+    && python -m pip install --no-cache-dir --upgrade pip pytest \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+# --- dave/ node deps (cached layer) ----------------------------------------
+COPY dave/package.json dave/package-lock.json ./dave/
+RUN cd dave && npm ci
+
+# --- app source ------------------------------------------------------------
+COPY wsai ./wsai
+COPY tests ./tests
+COPY dave ./dave
+COPY requirements.txt PLAN.md README.md ./
+COPY docker-entrypoint.sh /usr/local/bin/entrypoint
+RUN chmod +x /usr/local/bin/entrypoint
+
+# .env (DISCORD_BOT_TOKEN) is NOT baked in — mount it at runtime:
+#   -v $(pwd)/.env:/app/.env:ro
+ENTRYPOINT ["entrypoint"]
+CMD ["smoke"]

PLAN.md

@@ -0,0 +1,84 @@
+# watch_sceen_ai — 착수 계획 (확정본)
+
+디스코드 화면공유를 실시간으로 함께 보며 **음성으로** 대화하는 AI.
+
+## 방향 전환 (2026-08-11)
+
+- **화면공유(눈) 부분은 일단 보류**하고, 먼저 **음성 대화 루프 STT → 두뇌 → TTS**를 완성한다.
+- 파이프라인은 이제 눈 없이(source/vision = None) 돌아간다: `python -m wsai --voice`.
+- 화면공유 관문 1(DAVE) 성과는 아래에 보존. 나중에 눈을 다시 붙일 때 재사용한다.
+
+## 공식 봇 전환 (2026-08-16)
+
+- 보이스 접속(=STT 입력 경로)을 **셀프봇(유저 토큰) → 공식 Discord 봇**으로 마이그레이션 완료.
+  `dave/bot.mjs`(discord.js 14 + @discordjs/voice 0.19)가 봇 토큰으로 대상 채널 join →
+  DAVE/MLS E2EE Ready → 유저별 Opus 수신까지 라이브 검증됨. ToS-safe.
+- 셀프봇은 공식 봇이 막는 **화면공유 비디오 수신 트랙**용으로만 남기고, 그건 보류 상태다.
+  아래 1단계(눈) 셀프봇 서술은 그 보류된 비디오 트랙 기준이다.
+
+## 확정된 결정 (2026-08-09)
+
+1. **캡처 방식 (눈)** — 사용자 요구: 스크린샷/브라우저 캡처 ❌,
+   **실제 실시간 화면공유 스트림 자체를 진짜 클라이언트처럼 수신**해야 함.
+   → **셀프봇(유저 토큰)으로 프로토콜 레벨에서 비디오 RTP 스트림을 수신·디코딩**한다.
+   - 근거(원문 확인): Discord 공식 봇은 비디오를 완전 차단 → 봇으로는 불가. 반드시 유저 토큰(셀프봇) 필요.
+   - `discord.js-selfbot-v13`(werift-rtp 사용)이 수신 노출: `VoiceReceiver`(수신 스트림 레코딩), `StreamConnection`(스크린셰어), `receiverData` 이벤트(ssrcData: userId, **hasVideo**, RtpPacket). → 남의 화면공유 수신 기술적으로 가능.
+   - 수신 파이프: 셀프봇 join + Go Live "watch stream" 시그널링으로 비디오 SSRC 구독 → 암호화 RTP 비디오 수신 → 복호화(xsalsa20_poly1305/libsodium) → depayload(VP8/H264) → ffmpeg 디코드 → 프레임.
+   - 위험/제약: (a) 유저 토큰 = **Discord ToS 위반, 계정 밴 위험** → 반드시 버리는 계정 사용. (b) 완성형 "수신→프레임" 라이브러리는 없음, 조립 필요. (c) 주력 셀프봇 라이브러리는 2025-10 아카이브(read-only) → 프로토콜 변경 시 깨질 수 있음.
+   - ❌ 폐기: Xvfb + 브라우저 캡처(스크린샷) 방식 — 사용자가 명시적으로 거부. `poc/`는 참고용 폴백으로만 남김.
+2. **대화 방식** — 음성 실시간. STT→LLM→TTS 스트리밍으로 지연 최소화, 사용자 인터럽트 지원.
+3. **두뇌** — Claude **OAuth**로 가장 싸고 빠른 모델(Haiku 계열).
+   + .9의 RTX 5050으로 소형 로컬 VLM을 돌려 매 프레임 1차 이해·**화면 변화 감지**를 값싸게 처리,
+   어려운 화면만 Haiku로 에스컬레이션 (하이브리드).
+4. **실행 위치** — 이 리눅스 호스트(.9).
+
+## 단계별 착수 목록
+
+### 1단계 · 눈: 화면공유 스트림 수신  ← 지금
+- [x] 호스트 도구 확인 (Node 22/ffmpeg/libsodium 가능)
+- [x] 수신 방법 조사 확정: 셀프봇 프로토콜 레벨 비디오 RTP 수신
+- [x] **버리는 디스코드 유저 계정 + 토큰 확보, ToS/밴 위험 수용 확인** (`.env`, burner tkrmagid_bot)
+- [x] **[관문 1 통과] DAVE/MLS(E2EE) 게이트웨이 4017 검증** — 아래 참조
+- [ ] 셀프봇으로 음성채널 join → 상대 Go Live 스트림 watch/구독 (비디오 SSRC 획득)
+- [ ] 들어오는 비디오 RTP 수신 → **DAVE 복호화(daveSession.decrypt)** → depayload(VP8/H264)
+- [ ] ffmpeg로 디코드 → 프레임(JPEG/PNG) 추출 PoC (진짜 공유화면 수신 검증)
+- [ ] 프레임 변화 감지(동일 화면 반복 전송 방지) + 소스 인터페이스
+
+#### 관문 1 (arbiter 지정 fail-fast) — 통과 확정 2026-08-09
+DAVE(MLS) E2EE가 전면 강제라 "토큰만 꽂으면 됨"이 깨졌다는 게 핵심 리스크였음.
+`dave/gate.mjs` 로 실측 검증 → **통과**:
+- 셀프봇 유저토큰으로 메인 GW IDENTIFY → 음성채널 join → 보이스 GW v8 IDENTIFY(`max_dave_protocol_version=1`).
+- 결과: **close 4017 없음**. `SESSION_DESCRIPTION`에서 `dave_protocol_version=1` 협상됨(= 해당 채널 E2EE 활성).
+- `@snazzah/davey`(Rust NAPI, DAVE 구현)로 MLS 멤버십 핸드셰이크 완주:
+  op25 external_sender 수신 → op26 key_package 송신 → op27 proposals 처리(commit+welcome 생성)
+  → op28 commit_welcome 송신 → op29 announce_commit → `processCommit` → **MLS session ready=true**.
+- 5초간 서버 disconnect 없이 멤버십 유지, voicePrivacyCode 산출됨(실제 E2EE 그룹 참여 증명).
+- 결론: 셀프봇이 DAVE E2EE 보이스 그룹에 **정식 멤버로 합류 가능**. 옵션 A(직접 수신) 실현 가능 확정.
+- 바이너리 프레이밍 확정: 수신 `[u16 seq][u8 op][payload]`; op29/30 payload는 `[u16 transition_id][blob]`;
+  op27 payload는 `[u8 op_type][proposals]`; 송신 op26/28은 `[u8 op][blob(+welcome)]`.
+  processProposals에는 채널 내 인식 유저ID(op11 clients_connect)를 넘겨야 함(안 넘기면 UnexpectedUser).
+
+#### 다음 (관문 통과 후 A 강행)
+- 상대의 Go Live 스트림 구독: op **video** 스트림 SSRC 확보(스트림 시청 시그널링) — Go Live 중인 소스 필요.
+- UDP로 들어오는 SRTP 비디오 패킷 수신 → RTP 헤더 파싱 → `daveSession.decrypt(userId, VIDEO, packet)` E2EE 복호화 →
+  전송암호(aead_aes256_gcm_rtpsize) 복호 → VP8/H264 depayload → ffmpeg 디코드 → 프레임.
+
+### 3단계 · 두뇌 (하이브리드 비전)
+- [ ] 로컬 VLM(예: moondream2 / Qwen2-VL-2B) 프레임 1차 이해 + 변화 감지
+- [ ] Claude OAuth(Haiku)로 어려운 프레임 에스컬레이션
+- [ ] 최신 화면 맥락 저장소
+
+### 4단계 · 음성 대화
+- [ ] STT (faster-whisper, 스트리밍/부분전사)
+- [ ] 대화 브레인 (화면맥락 + 대화이력)
+- [ ] TTS (MeloTTS 등, 이 호스트에 자산 있음)
+- [ ] 말 끊기 인터럽트 + 지연 최적화
+
+### 5단계 · 통합·운영
+- [ ] 인지 루프 + 대화 루프 동시 실행
+- [ ] proactive(화면 크게 바뀌면 먼저 말 걸기)
+- [ ] 끊김 복구, 비용 모니터링, 서비스화
+
+## 주의
+- 캡처용 디스코드 계정은 메인 말고 **별도 계정 권장**(자동화 회색지대, 밴 위험 최소화).
+- 로컬 GPU 8GB VRAM 제약 → 소형 VLM만. 큰 이해는 클라우드.

README.md

@@ -8,8 +8,9 @@ STT → 두뇌 → TTS부터 완성**하는 단계다.
 - 현재 상태: 음성 루프 뼈대 동작 — 파이프라인이 **눈 없이(화면공유 없이)** 돌아간다
   (`python -m wsai --voice`). STT/TTS/두뇌는 아직 mock이며, 실제 엔진 연결이 다음 목표.
 - 보류 중: 화면공유 **비디오** 수신(눈). 단, STT 입력은 **디스코드 보이스로 유저 음성을
-  수신**하므로 셀프봇의 보이스 접속 자체는 지금도 쓴다(비디오만 미룸). E2EE 합류 관문은
+  수신**하므로 보이스 접속 자체는 지금도 쓴다(비디오만 미룸). 이 보이스 접속은
-  이미 통과해 두었다(아래 8장).
+  **공식 Discord 봇**(`dave/bot.mjs`, discord.js + @discordjs/voice)으로 하며 DAVE/MLS
+  E2EE 합류·수신까지 라이브로 검증됐다(아래 8장). 셀프봇 경로는 폐기(비디오 트랙용만 보존).
 
 ---
 
@@ -53,9 +54,10 @@ STT → 두뇌 → TTS부터 완성**하는 단계다.
 ## 3. 귀 / 두뇌 / 입 — 지금 만드는 음성 루프
 
 ### 귀 (STT) — 디스코드 보이스 수신
-- 로컬 마이크가 아니라 **디스코드 보이스에서 유저가 말하는 음성을 수신**한다. 셀프봇이
+- 로컬 마이크가 아니라 **디스코드 보이스에서 유저가 말하는 음성을 수신**한다. **공식 봇**이
-  보이스 채널에 접속해(이미 통과한 DAVE/MLS E2EE 경로) 유저의 Opus 오디오 RTP를 받고,
+  보이스 채널에 접속해(이미 통과한 DAVE/MLS E2EE 경로) 유저의 Opus 오디오를 받는다.
-  복호 → Opus 디코드 → PCM으로 만들어 STT에 흘려보낸다.
+  @discordjs/voice의 `VoiceReceiver`가 DAVE 복호까지 처리해 유저별 Opus 스트림을 주고,
+  이를 Opus 디코드 → PCM으로 만들어 STT에 흘려보낸다.
 - `faster-whisper`로 실시간 부분 전사를 계속 돌리고, VAD로 발화 종료(endpointing)를 잡는다.
   종료 판정 시점엔 전사가 사실상 끝나 있어, 판정 후 남는 건 VAD hangover + 짧은 마지막
   디코드뿐이다. 목표 종료→텍스트 확정 ~150ms.
@@ -128,12 +130,15 @@ ffmpeg 디코드 → 프레임. DAVE는 `@snazzah/davey`(Rust NAPI), 수신은 `
 
 ```bash
 .venv/bin/python -m wsai --voice    # 눈 없는 음성 루프 데모 (STT→두뇌→TTS, 지금 초점)
+.venv/bin/python -m wsai --dashboard # 상태 사이트 + 짧은 mock 샘플 후 대기(무한 생성 안 함)
 .venv/bin/python -m wsai            # mock 데모 (눈 포함 전체 흐름, 몇 프레임 돌고 종료)
 .venv/bin/python -m wsai --env      # WSAI_* 환경변수로 백엔드 조립
 .venv/bin/python -m pip install pytest && .venv/bin/python -m pytest -q   # 스모크 테스트
 ```
 
 - `WSAI_SOURCE=none WSAI_VISION=none` 으로도 눈 없이(음성 루프만) 조립할 수 있다.
+- `--dashboard`는 기본으로 mock 발화 3개만 만든 뒤 대기한다. UI 시연용으로 계속 만들고 싶을 때만
+  `--dashboard-loop-demo`를 추가한다.
 - 백엔드별 추가 설치는 `requirements.txt` 주석 참고(faster-whisper, mss/pillow, anthropic 등).
 
 ---
@@ -149,7 +154,8 @@ ffmpeg 디코드 → 프레임. DAVE는 `@snazzah/davey`(Rust NAPI), 수신은 `
 | `wsai/backends/mock.py` | 무의존성 mock 전 계열 |
 | `wsai/backends/capture_mss.py` | 로컬 화면 캡처(트랙 B, 보류) |
 | `wsai/backends/claude.py` | Claude 비전 + 두뇌 |
-| `dave/gate.mjs` | DAVE/MLS E2EE 합류 관문 검증(통과 증거, 눈 재개 시 재사용) |
+| `dave/bot.mjs` | **공식 봇** 보이스 접속 + 유저별 Opus 수신(DAVE E2EE, 현재 경로) |
+| `dave/gate.mjs`, `dave/join.mjs` | 레거시 셀프봇 경로(폐기, 비디오 트랙 재개 시 참고용) |
 | `poc/` | Xvfb+Chromium 캡처 실험(트랙 B 참고) |
 | `PLAN.md` | 단계별 착수 계획 |
 | `tests/test_pipeline.py` | 파이프라인 스모크 테스트 |
@@ -165,8 +171,9 @@ ffmpeg 디코드 → 프레임. DAVE는 `@snazzah/davey`(Rust NAPI), 수신은 `
       · 검증: `python -m wsai --voice`가 화면 없이 발화마다 응답을 낸다(테스트 포함).
 - [ ] **V1 · 진짜 TTS(입).** MeloTTS로 두뇌 응답을 실제 음성으로 합성.
       · 검증: 응답 텍스트가 .wav로 합성돼 들린다.
-- [ ] **V2 · 진짜 STT(귀).** faster-whisper로 오디오를 텍스트로 전사.
+- [x] **V2 · 진짜 STT(귀) — 엔진 완성.** faster-whisper(상주 워커, `WSAI_STT=whisper`)로 wav를 한국어 텍스트로 전사.
-      · 검증: 오디오 파일 입력이 텍스트로 나온다(이후 디스코드 보이스 수신 오디오로 연결).
+      · 검증: 실제 왕복(MeloTTS wav → whisper)에서 문장이 거의 그대로 복원됨. warm 전사 ~1.2s(CPU, small/int8).
+      · 남은 것: 디스코드 보이스 수신 오디오(`audio_source`)를 붙여 실시간 발화 스트림으로 연결(V4).
 - [ ] **V3 · 진짜 두뇌.** Claude OAuth(Haiku)로 대화 응답 생성.
       · 검증: 실제 발화에 자연스러운 답이 나온다.
 - [ ] **V4 · 음성 왕복 + barge-in.** 디스코드 보이스 수신→STT→Brain→TTS 스트리밍, 말 끊기, 지연 측정.
@@ -183,17 +190,25 @@ ffmpeg 디코드 → 프레임. DAVE는 `@snazzah/davey`(Rust NAPI), 수신은 `
   첫 소리 지연을 GPU에서 실측해, 1초 예산에 맞는 가장 자연스러운 엔진을 채택한다(측정 기반 선택).
 - **두뇌** — Claude **OAuth**, 이 호스트에 연결된 것과 **동일 크리덴셜 공유**
   (`/home/claude/EJClaw/data/claude/.credentials.json`). API키 아님.
-- **STT 입력 경로** — 디스코드 보이스 수신(셀프봇). 유저 음성 Opus RTP를 받아 DAVE 복호·디코드해 STT로.
+- **STT 입력 경로** — 디스코드 보이스 수신(**공식 봇**). 유저 음성 Opus를 DAVE 복호·디코드해 STT로.
+
+### 공식 봇 전환 (2026-08-16)
+- 보이스 접속을 셀프봇(유저 토큰)에서 **공식 Discord 봇**으로 마이그레이션. `dave/bot.mjs`
+  (discord.js 14 + @discordjs/voice 0.19)가 봇 토큰으로 로그인 → 대상 채널 join →
+  DAVE/MLS E2EE Ready → `VoiceReceiver`로 유저별 Opus 수신 → PCM 디코드까지 라이브 검증됨.
+- 봇은 이미 대상 서버(사지방)에 초대돼 있어 추가 초대 조치 불필요. 미초대 시
+  `node dave/bot.mjs --invite`로 초대 URL을 출력한다.
+- 셀프봇이 필요한 건 공식 봇이 막힌 **화면공유 비디오 수신**뿐이며, 그건 보류 상태다.
 
 ### 배포/테스트 목표
 - 대상: 디스코드 서버 `1352269198297923648`의 보이스 채널 `1352269198914621465`.
-- 유저봇(셀프봇) 토큰: `.env`의 `DISCORD_SELFBOT_TOKEN`(버너 계정, 밴 리스크 수용).
+- 봇 토큰: `.env`의 `DISCORD_BOT_TOKEN`(테스트봇, app id `1538122882528321536`). ToS-safe.
 - .9 로컬 GPU 도커 이미지로 올려, 그 채널에 접속한 뒤 사람이 말하면 대화하도록 한다.
 - 첫 로딩 워밍업: 시작 시 모델 프리로드 + 더미 추론(CUDA 워밍) + 보이스 미리 접속 → 첫 대화도 지연 최소.
 
 ### 구현 마일스톤
-- **M1** 셀프봇이 대상 보이스 채널에 상주 접속(DAVE 통과) + 발화자(SSRC) 감지  ← 지금
+- **M1** ✅ **공식 봇**이 대상 보이스 채널에 상주 접속(DAVE 통과) + 발화자 감지 — `dave/bot.mjs`로 라이브 검증
-- **M2** 유저 음성 Opus RTP 수신 → DAVE 복호 → PCM (라이브 발화자 필요, 최고 난이도)
+- **M2** 유저 음성 Opus 수신 → DAVE 복호 → PCM — @discordjs/voice `VoiceReceiver`가 대부분 처리(라이브 발화자로 최종 검증만 남음)
 - **M3** faster-whisper STT(부분전사+VAD) → **M4** Claude OAuth(Haiku) 두뇌
 - **M5** 한국어 TTS 첫 구절 청크를 보이스로 송신(DAVE 암호화) + barge-in
 - **M6** 통합 + 워밍업 + .9 GPU 도커 이미지화, 채널 라이브 테스트

dave/bot.mjs

@@ -0,0 +1,249 @@
+// Official Discord BOT voice joiner (replaces the selfbot join.mjs).
+//
+// Why this exists: the project migrated off the user-token selfbot path onto an
+// official bot application. For the voice loop (STT input) this is the fully
+// supported, ToS-safe path — @discordjs/voice lets a bot JOIN a voice channel,
+// pass the DAVE/MLS E2EE handshake (via @snazzah/davey, handled internally), and
+// RECEIVE per-user Opus audio. Only screenshare VIDEO receive still requires a
+// selfbot, and video ("눈") is deferred, so nothing here needs a user token.
+//
+// This replicates M1 (join target channel, stay, detect speakers) and, for free,
+// gives partial M2: it decodes each speaker's Opus to PCM and counts frames — the
+// exact stream the STT stage will consume.
+//
+// Usage:
+//   node bot.mjs                 # join and stay until killed
+//   RUN_MS=15000 node bot.mjs    # join, hold 15s, then leave (for verification)
+//
+// Requires: the bot must be INVITED to the target guild with the "Connect" and
+// "Speak" voice permissions. See dave/README or run `node bot.mjs --invite`.
+
+import fs from 'node:fs';
+import { Readable } from 'node:stream';
+import {
+  Client,
+  GatewayIntentBits,
+} from 'discord.js';
+import {
+  joinVoiceChannel,
+  getVoiceConnection,
+  entersState,
+  VoiceConnectionStatus,
+  EndBehaviorType,
+  createAudioPlayer,
+  createAudioResource,
+  StreamType,
+  NoSubscriberBehavior,
+} from '@discordjs/voice';
+import prism from 'prism-media';
+
+// ---------- config ----------
+function loadEnvFile() {
+  try {
+    return Object.fromEntries(
+      fs.readFileSync(new URL('../.env', import.meta.url), 'utf8')
+        .split('\n').filter(l => l && !l.startsWith('#') && l.includes('='))
+        .map(l => { const i = l.indexOf('='); return [l.slice(0, i).trim(), l.slice(i + 1).trim()]; })
+    );
+  } catch { return {}; }
+}
+const env = loadEnvFile();
+const TOKEN = process.env.DISCORD_BOT_TOKEN || env.DISCORD_BOT_TOKEN;
+const GUILD_ID = process.env.GUILD_ID || env.GUILD_ID || '1352269198297923648';
+const CHANNEL_ID = process.env.CHANNEL_ID || env.CHANNEL_ID || '1352269198914621465';
+const RUN_MS = process.env.RUN_MS != null ? Number(process.env.RUN_MS) : 0; // 0 = stay forever
+// Python STT+TTS voice-turn endpoint (run `python -m wsai --voice-server`).
+const VOICE_ENDPOINT = process.env.WSAI_VOICE_ENDPOINT || env.WSAI_VOICE_ENDPOINT
+  || 'http://127.0.0.1:8787/api/voice-turn';
+// Ignore utterances shorter than this many PCM bytes (48kHz*2ch*2B = 192000 B/s),
+// so key clicks / brief noise don't trigger a turn. ~0.35s.
+const MIN_UTTERANCE_BYTES = Number(process.env.WSAI_MIN_UTTERANCE_BYTES || 67000);
+
+const t0 = Date.now();
+const log = (...a) => console.log(`[+${String(Date.now() - t0).padStart(6)}ms]`, ...a);
+
+// Throttle bursty repeated logs. DAVE (E2EE) group transitions — someone joins
+// or leaves the voice channel — briefly deliver undecryptable packets, so the
+// same "recv stream error" can fire many times in a second. Log the first
+// occurrence of a given message immediately, then collapse repeats within a
+// window into one summary line instead of flooding the log.
+const _throttle = new Map(); // key -> { count, timer }
+function logThrottled(key, msg, windowMs = 10_000) {
+  const e = _throttle.get(key);
+  if (e) { e.count++; return; }
+  log(msg);
+  const timer = setTimeout(() => {
+    const cur = _throttle.get(key);
+    _throttle.delete(key);
+    if (cur && cur.count > 0) log(`${msg} (+${cur.count} more in ${Math.round(windowMs / 1000)}s)`);
+  }, windowMs);
+  if (typeof timer.unref === 'function') timer.unref();
+  _throttle.set(key, { count: 0, timer });
+}
+
+// PCM s16le -> WAV container (so the Python side can ffmpeg-decode it).
+function wavHeader(dataLen, sampleRate = 48000, channels = 2, bits = 16) {
+  const blockAlign = channels * bits / 8;
+  const b = Buffer.alloc(44);
+  b.write('RIFF', 0); b.writeUInt32LE(36 + dataLen, 4); b.write('WAVE', 8);
+  b.write('fmt ', 12); b.writeUInt32LE(16, 16); b.writeUInt16LE(1, 20);
+  b.writeUInt16LE(channels, 22); b.writeUInt32LE(sampleRate, 24);
+  b.writeUInt32LE(sampleRate * blockAlign, 28); b.writeUInt16LE(blockAlign, 32);
+  b.writeUInt16LE(bits, 34); b.write('data', 36); b.writeUInt32LE(dataLen, 40);
+  return b;
+}
+
+// Speak audio into the voice channel. Feed the reply wav bytes through ffmpeg
+// (StreamType.Arbitrary) so @discordjs/voice re-encodes to Opus.
+let voicePlayer = null;
+function playReply(wavBytes) {
+  if (!voicePlayer || !wavBytes || wavBytes.length === 0) return;
+  const resource = createAudioResource(Readable.from(wavBytes), { inputType: StreamType.Arbitrary });
+  voicePlayer.play(resource);
+}
+
+// One utterance: PCM -> WAV -> POST to Python -> play the reply back.
+async function handleUtterance(userId, pcm) {
+  if (pcm.length < MIN_UTTERANCE_BYTES) {
+    log(`utterance too short user=${userId} bytes=${pcm.length} — skip`);
+    return;
+  }
+  const wav = Buffer.concat([wavHeader(pcm.length), pcm]);
+  let resp;
+  try {
+    resp = await fetch(VOICE_ENDPOINT, {
+      method: 'POST', headers: { 'Content-Type': 'audio/wav' }, body: wav,
+    });
+  } catch (e) {
+    log(`voice-turn POST failed (is \`python -m wsai --voice-server\` running?): ${e.message}`);
+    return;
+  }
+  if (!resp.ok) { log(`voice-turn HTTP ${resp.status}`); return; }
+  const heard = decodeURIComponent(resp.headers.get('X-Heard') || '');
+  const reply = decodeURIComponent(resp.headers.get('X-Reply') || '');
+  const buf = Buffer.from(await resp.arrayBuffer());
+  log(`heard="${heard}" reply="${reply}" replyWav=${buf.length}B`);
+  playReply(buf);
+}
+
+// ---------- invite URL helper ----------
+// A bot cannot add itself to a server; a server admin must click an OAuth2 invite
+// URL once. Print it so the user can authorise the bot with voice permissions.
+if (process.argv.includes('--invite')) {
+  const APP_ID = process.env.DISCORD_APP_ID || env.DISCORD_APP_ID || '';
+  // permissions: Connect(1<<20) | Speak(1<<21) | UseVAD(1<<25) | ViewChannel(1<<10)
+  const perms = (1n << 20n) | (1n << 21n) | (1n << 25n) | (1n << 10n);
+  if (!APP_ID) { console.error('set DISCORD_APP_ID (application/client id) in .env to build the invite URL'); process.exit(2); }
+  console.log(`https://discord.com/oauth2/authorize?client_id=${APP_ID}&scope=bot&permissions=${perms}`);
+  process.exit(0);
+}
+
+if (!TOKEN) { console.error('no DISCORD_BOT_TOKEN in env or ../.env'); process.exit(2); }
+
+// ---------- client ----------
+const client = new Client({
+  intents: [GatewayIntentBits.Guilds, GatewayIntentBits.GuildVoiceStates],
+});
+
+const perUser = new Map(); // userId -> { opusPackets, pcmFrames }
+
+let leaving = false;
+function leaveAndExit(code = 0) {
+  if (leaving) return;
+  leaving = true;
+  try { getVoiceConnection(GUILD_ID)?.destroy(); } catch {}
+  const summary = [...perUser.entries()].map(([u, s]) => `${u}:opus=${s.opusPackets},pcm=${s.pcmFrames}`);
+  log(`leaving. speakers heard: ${summary.length ? summary.join(' ') : '(none)'}`);
+  try { client.destroy(); } catch {}
+  setTimeout(() => process.exit(code), 300);
+}
+process.on('SIGINT', () => { log('SIGINT'); leaveAndExit(0); });
+process.on('SIGTERM', () => { log('SIGTERM'); leaveAndExit(0); });
+
+// Hard time-box ceiling, armed at startup regardless of handshake state.
+if (RUN_MS > 0) setTimeout(() => { log(`RUN_MS=${RUN_MS} hard ceiling elapsed — leaving`); leaveAndExit(0); }, RUN_MS);
+
+client.once('clientReady', async () => {
+  log(`logged in as ${client.user.tag} (${client.user.id})`);
+  let guild, channel;
+  try {
+    guild = await client.guilds.fetch(GUILD_ID);
+    channel = await guild.channels.fetch(CHANNEL_ID);
+  } catch (e) {
+    log(`FATAL: cannot access guild/channel — is the bot invited to guild ${GUILD_ID}? (${e.message})`);
+    log('run `node bot.mjs --invite` and have a server admin authorise the bot, then retry.');
+    return leaveAndExit(1);
+  }
+  if (!channel || !channel.isVoiceBased()) { log(`FATAL: channel ${CHANNEL_ID} is not a voice channel`); return leaveAndExit(1); }
+
+  log(`joining voice: guild="${guild.name}" channel="${channel.name}"`);
+  const connection = joinVoiceChannel({
+    channelId: CHANNEL_ID,
+    guildId: GUILD_ID,
+    adapterCreator: guild.voiceAdapterCreator,
+    selfDeaf: false, // MUST be false to receive audio (the STT input path)
+    selfMute: false, // false so we can also speak later (M5 TTS)
+  });
+  connection.on('error', (e) => log(`voice connection error: ${e.message}`));
+
+  try {
+    // The DAVE/MLS handshake here cycles signalling<->connecting several times
+    // and can take ~25s, so give it a generous ceiling before declaring failure.
+    await entersState(connection, VoiceConnectionStatus.Ready, 40_000);
+  } catch (e) {
+    log(`FATAL: voice connection did not become Ready in 40s (${e.message})`);
+    return leaveAndExit(1);
+  }
+  log(`✅ JOINED & READY. channel=${CHANNEL_ID} — staying connected, listening for speakers…`);
+
+  // Playback path (bot speaks): one player, subscribed to the connection.
+  voicePlayer = createAudioPlayer({ behaviors: { noSubscriber: NoSubscriberBehavior.Play } });
+  voicePlayer.on('error', (e) => log(`player error: ${e.message}`));
+  connection.subscribe(voicePlayer);
+  log(`voice endpoint: ${VOICE_ENDPOINT}`);
+
+  // ---------- receive path: capture each utterance and run the voice turn ----
+  const receiver = connection.receiver;
+  const active = new Set(); // userIds with an in-flight subscription (avoid dupes)
+  receiver.speaking.on('start', (userId) => {
+    if (userId === client.user.id || active.has(userId)) return; // skip self / dupes
+    active.add(userId);
+    if (!perUser.has(userId)) perUser.set(userId, { opusPackets: 0, pcmFrames: 0 });
+    log(`SPEAKING start user=${userId}`);
+    const opusStream = receiver.subscribe(userId, {
+      // End the utterance after a short silence so natural pauses don't cut words.
+      end: { behavior: EndBehaviorType.AfterSilence, duration: 800 },
+    });
+    // Decode Opus -> 48kHz stereo s16le PCM, buffered until the utterance ends.
+    const decoder = new prism.opus.Decoder({ rate: 48000, channels: 2, frameSize: 960 });
+    const chunks = [];
+    opusStream.on('data', () => { perUser.get(userId).opusPackets++; });
+    // A receive-stream error (e.g. a DAVE decrypt/UDP GenericFailure on one
+    // packet) must NOT crash the process — log it and free the slot so the
+    // next utterance still works.
+    opusStream.on('error', (e) => { logThrottled(`recv:${e.message}`, `recv stream error user=${userId}: ${e.message}`); active.delete(userId); });
+    opusStream.pipe(decoder);
+    decoder.on('data', (d) => {
+      chunks.push(d);
+      const s = perUser.get(userId); s.pcmFrames++;
+    });
+    decoder.on('error', (e) => log(`decode error user=${userId}: ${e.message}`));
+    decoder.on('end', () => {
+      active.delete(userId);
+      const pcm = Buffer.concat(chunks);
+      log(`utterance end user=${userId} pcm=${pcm.length}B — running voice turn`);
+      handleUtterance(userId, pcm).catch((e) => log(`voice turn error: ${e.message}`));
+    });
+  });
+
+  connection.on(VoiceConnectionStatus.Disconnected, () => {
+    log('voice: disconnected — attempting to resume…');
+    Promise.race([
+      entersState(connection, VoiceConnectionStatus.Signalling, 5_000),
+      entersState(connection, VoiceConnectionStatus.Connecting, 5_000),
+    ]).catch(() => { log('voice: could not resume, leaving'); leaveAndExit(0); });
+  });
+});
+
+client.on('error', (e) => log('client error', e.message));
+client.login(TOKEN).catch((e) => { log(`FATAL: login failed — ${e.message}`); process.exit(1); });

dave/gate.mjs

@@ -0,0 +1,321 @@
+// Fail-fast gate (arbiter checkpoint #1):
+// Can a selfbot pass the Discord voice DAVE/MLS handshake (no close 4017),
+// and does the voice gateway negotiate DAVE + push the external-sender (op25)?
+//
+// This intentionally stops at "gate evidence": we log the voice IDENTIFY result,
+// whether close code 4017 occurs, the negotiated dave_protocol_version, and any
+// DAVE opcodes the server pushes. It joins a voice channel muted+deaf and leaves
+// immediately after collecting evidence. No media is transmitted.
+//
+// Usage: DAVE_VER=1 node gate.mjs   (declare DAVE support)
+//        DAVE_VER=0 node gate.mjs   (declare DAVE UNsupported -> expect rejection)
+
+import WebSocket from 'ws';
+import dgram from 'node:dgram';
+import fs from 'node:fs';
+import * as davey from '@snazzah/davey';
+
+// --- config ---
+const env = Object.fromEntries(
+  fs.readFileSync(new URL('../.env', import.meta.url), 'utf8')
+    .split('\n').filter(l => l && !l.startsWith('#') && l.includes('='))
+    .map(l => { const i = l.indexOf('='); return [l.slice(0, i).trim(), l.slice(i + 1).trim()]; })
+);
+const TOKEN = env.DISCORD_SELFBOT_TOKEN;
+const GUILD_ID = process.env.GUILD_ID || '1352269198297923648';
+const CHANNEL_ID = process.env.CHANNEL_ID || '1352269198914621465'; // "일반"
+const DAVE_VER = process.env.DAVE_VER != null ? Number(process.env.DAVE_VER) : davey.DAVE_PROTOCOL_VERSION;
+const HARD_TIMEOUT_MS = Number(process.env.TIMEOUT_MS || 30000);
+
+if (!TOKEN) { console.error('no token'); process.exit(2); }
+
+const t0 = Date.now();
+const log = (...a) => console.log(`[+${String(Date.now() - t0).padStart(6)}ms]`, ...a);
+console.log(`davey VERSION=${davey.VERSION} DAVE_PROTOCOL_VERSION=${davey.DAVE_PROTOCOL_VERSION}`);
+log(`gate start: declaring max_dave_protocol_version=${DAVE_VER}, channel=${CHANNEL_ID}`);
+
+const evidence = {
+  daveVerDeclared: DAVE_VER,
+  mainReady: false,
+  voiceServerReceived: false,
+  voiceIdentifySent: false,
+  voiceReady: false,
+  voiceReadyModes: null,
+  negotiatedDaveVersion: null,
+  sawExternalSender: false,
+  sawDaveOpcodes: [],
+  mlsSessionReady: false,
+  voiceCloseCode: null,
+  voiceCloseReason: null,
+  verdict: 'INCOMPLETE',
+  notes: [],
+};
+
+let mainWs, voiceWs, udp;
+let daveSession = null;
+let voiceState = { session_id: null, token: null, endpoint: null, ssrc: null, ip: null, port: null, mode: null };
+let finished = false;
+
+function finish(verdict, extra) {
+  if (finished) return;
+  finished = true;
+  evidence.verdict = verdict;
+  if (extra) evidence.notes.push(extra);
+  // leave the voice channel politely
+  try { mainWs?.send(JSON.stringify({ op: 4, d: { guild_id: GUILD_ID, channel_id: null, self_mute: true, self_deaf: true } })); } catch {}
+  setTimeout(() => {
+    try { voiceWs?.close(); } catch {}
+    try { udp?.close(); } catch {}
+    try { mainWs?.close(); } catch {}
+    console.log('\n===== GATE EVIDENCE =====');
+    console.log(JSON.stringify(evidence, null, 2));
+    process.exit(0);
+  }, 400);
+}
+setTimeout(() => finish(evidence.verdict === 'INCOMPLETE' ? 'TIMEOUT' : evidence.verdict, `hard timeout ${HARD_TIMEOUT_MS}ms`), HARD_TIMEOUT_MS);
+
+// ---------- MAIN GATEWAY ----------
+mainWs = new WebSocket('wss://gateway.discord.gg/?v=10&encoding=json');
+let mainHb;
+mainWs.on('open', () => log('main gw: open'));
+mainWs.on('message', (raw) => {
+  const p = JSON.parse(raw.toString());
+  if (p.op === 10) {
+    const iv = p.d.heartbeat_interval;
+    mainHb = setInterval(() => { try { mainWs.send(JSON.stringify({ op: 1, d: null })); } catch {} }, iv);
+    mainWs.send(JSON.stringify({ op: 2, d: {
+      token: TOKEN,
+      capabilities: 16381,
+      properties: { os: 'Linux', browser: 'Chrome', device: '', system_locale: 'en-US', browser_user_agent: 'Mozilla/5.0', browser_version: '124.0', os_version: '', release_channel: 'stable', client_build_number: 300000 },
+      compress: false,
+      presence: { status: 'invisible', since: 0, activities: [], afk: false },
+    }}));
+    log('main gw: sent IDENTIFY');
+  } else if (p.op === 0) {
+    if (p.t === 'READY') {
+      evidence.mainReady = true;
+      log(`main gw: READY as ${p.d.user?.username} (${p.d.user?.id})`);
+      // join voice channel muted+deaf
+      mainWs.send(JSON.stringify({ op: 4, d: { guild_id: GUILD_ID, channel_id: CHANNEL_ID, self_mute: true, self_deaf: true } }));
+      log('main gw: sent Voice State Update (join)');
+    } else if (p.t === 'VOICE_STATE_UPDATE' && p.d.user_id) {
+      if (p.d.session_id) { voiceState.session_id = p.d.session_id; log('VOICE_STATE_UPDATE session_id acquired'); maybeConnectVoice(); }
+    } else if (p.t === 'VOICE_SERVER_UPDATE') {
+      voiceState.token = p.d.token; voiceState.endpoint = p.d.endpoint;
+      evidence.voiceServerReceived = true;
+      log(`VOICE_SERVER_UPDATE endpoint=${p.d.endpoint}`);
+      maybeConnectVoice();
+    }
+  }
+});
+mainWs.on('close', (c, r) => { log(`main gw: close ${c} ${r}`); clearInterval(mainHb); });
+mainWs.on('error', (e) => log('main gw error', e.message));
+
+// ---------- VOICE GATEWAY ----------
+function maybeConnectVoice() {
+  if (voiceWs || !voiceState.session_id || !voiceState.token || !voiceState.endpoint) return;
+  const url = `wss://${voiceState.endpoint}/?v=8`;
+  log(`voice gw: connecting ${url}`);
+  voiceWs = new WebSocket(url);
+  let voiceHb, lastSeq = null;
+  const knownUsers = new Set(['1513862586112671786']);
+  voiceWs.on('open', () => {
+    voiceWs.send(JSON.stringify({ op: 0, d: {
+      server_id: GUILD_ID,
+      user_id: '1513862586112671786',
+      session_id: voiceState.session_id,
+      token: voiceState.token,
+      max_dave_protocol_version: DAVE_VER,
+    }}));
+    evidence.voiceIdentifySent = true;
+    log(`voice gw: sent IDENTIFY (max_dave_protocol_version=${DAVE_VER})`);
+  });
+  voiceWs.on('message', (raw, isBinary) => {
+    if (isBinary) {
+      const buf = Buffer.isBuffer(raw) ? raw : Buffer.from(raw);
+      // received binary DAVE frame: uint16 seq BE, uint8 opcode, payload
+      const seq = buf.readUInt16BE(0);
+      const op = buf.readUInt8(2);
+      const payload = buf.subarray(3);
+      lastSeq = seq;
+      onDaveBinary(op, payload, seq);
+      return;
+    }
+    const p = JSON.parse(raw.toString());
+    handleVoiceJson(p);
+  });
+  voiceWs.on('close', (c, r) => {
+    evidence.voiceCloseCode = c; evidence.voiceCloseReason = r?.toString() || '';
+    log(`voice gw: CLOSE code=${c} reason="${evidence.voiceCloseReason}"`);
+    clearInterval(voiceHb);
+    if (c === 4017) finish('FAIL_4017', 'voice gateway rejected with close 4017 (DAVE-unsupported/protocol)');
+    else if (!finished) finish(evidence.voiceReady ? evidence.verdict : 'FAIL_VOICE_CLOSED', `voice closed ${c}`);
+  });
+  voiceWs.on('error', (e) => log('voice gw error', e.message));
+
+  function handleVoiceJson(p) {
+    switch (p.op) {
+      case 8: { // HELLO
+        const iv = p.d.heartbeat_interval;
+        voiceHb = setInterval(() => {
+          try { voiceWs.send(JSON.stringify({ op: 3, d: { t: Date.now(), seq_ack: lastSeq ?? 0 } })); } catch {}
+        }, iv);
+        log(`voice gw: HELLO heartbeat_interval=${iv}`);
+        break;
+      }
+      case 2: { // READY
+        evidence.voiceReady = true;
+        evidence.voiceReadyModes = p.d.modes;
+        voiceState.ssrc = p.d.ssrc; voiceState.ip = p.d.ip; voiceState.port = p.d.port;
+        log(`voice gw: READY ssrc=${p.d.ssrc} udp=${p.d.ip}:${p.d.port} modes=${JSON.stringify(p.d.modes)}`);
+        doUdpDiscoveryAndSelect(p.d);
+        break;
+      }
+      case 4: { // Session Description / select_protocol_ack (carries dave_protocol_version)
+        if (p.d && p.d.dave_protocol_version != null) {
+          evidence.negotiatedDaveVersion = p.d.dave_protocol_version;
+          log(`voice gw: SESSION_DESCRIPTION dave_protocol_version=${p.d.dave_protocol_version} mode=${p.d.mode}`);
+        } else {
+          log(`voice gw: SESSION_DESCRIPTION (no dave version field) mode=${p.d?.mode}`);
+        }
+        evaluateGate();
+        break;
+      }
+      case 11: { // clients connect
+        for (const u of (p.d.user_ids || [])) knownUsers.add(u);
+        log(`voice gw: op11 clients_connect ${JSON.stringify(p.d.user_ids)}`);
+        break;
+      }
+      case 20: { if (p.d.user_id) knownUsers.add(p.d.user_id); log(`voice gw: op20 platform user=${p.d.user_id}`); break; }
+      case 21: { // dave_prepare_transition (JSON)
+        evidence.sawDaveOpcodes.push(21);
+        log(`voice gw: DAVE op21 prepare_transition ${JSON.stringify(p.d)}`);
+        // ack readiness
+        try { voiceWs.send(JSON.stringify({ op: 23, d: { transition_id: p.d.transition_id } })); } catch {}
+        break;
+      }
+      case 22: {
+        evidence.sawDaveOpcodes.push(22);
+        log(`voice gw: DAVE op22 execute_transition ${JSON.stringify(p.d)}`);
+        if (daveSession && daveSession.ready) { evidence.mlsSessionReady = true; }
+        evaluateGate();
+        break;
+      }
+      case 24: {
+        evidence.sawDaveOpcodes.push(24);
+        log(`voice gw: DAVE op24 prepare_epoch ${JSON.stringify(p.d)}`);
+        break;
+      }
+      case 31: {
+        evidence.sawDaveOpcodes.push(31);
+        log(`voice gw: DAVE op31 invalid_commit_welcome ${JSON.stringify(p.d)}`);
+        break;
+      }
+      default:
+        log(`voice gw: op${p.op} ${JSON.stringify(p.d)?.slice(0, 200)}`);
+    }
+  }
+
+  function onDaveBinary(op, payload, seq) {
+    evidence.sawDaveOpcodes.push(op);
+    log(`voice gw: DAVE binary op${op} seq=${seq} len=${payload.length}`);
+    try {
+      switch (op) {
+        case 25: { // external sender package
+          evidence.sawExternalSender = true;
+          if (!daveSession) {
+            daveSession = new davey.DAVESession(evidence.negotiatedDaveVersion || DAVE_VER, '1513862586112671786', CHANNEL_ID);
+          }
+          daveSession.setExternalSender(payload);
+          const kp = daveSession.getSerializedKeyPackage();
+          // send op26 key package (binary): [uint8 opcode][payload]
+          voiceWs.send(Buffer.concat([Buffer.from([26]), kp]), { binary: true });
+          log(`voice gw: sent DAVE op26 key_package len=${kp.length}`);
+          break;
+        }
+        case 27: { // proposals: [operation_type u8][proposals...]
+          const opType = payload.readUInt8(0); // ProposalsOperationType
+          const proposals = payload.subarray(1);
+          const known = Array.from(knownUsers);
+          const res = daveSession.processProposals(opType, proposals, known);
+          if (res && res.commit) {
+            const parts = [Buffer.from([28]), res.commit];
+            if (res.welcome) parts.push(res.welcome);
+            voiceWs.send(Buffer.concat(parts), { binary: true });
+            log(`voice gw: sent DAVE op28 commit_welcome commit=${res.commit.length} welcome=${res.welcome?.length || 0} (known=${known.length})`);
+          } else {
+            log(`voice gw: op27 processed, no commit produced (known=${known.length})`);
+          }
+          break;
+        }
+        case 29: { // announce commit transition: [transition_id u16][commit...]
+          const commit = payload.subarray(2);
+          daveSession.processCommit(commit);
+          if (daveSession.ready) evidence.mlsSessionReady = true;
+          log(`voice gw: processed op29 commit (tid=${payload.readUInt16BE(0)}), sessionReady=${daveSession.ready}`);
+          evaluateGate();
+          break;
+        }
+        case 30: { // welcome: [transition_id u16][welcome...]
+          const welcome = payload.subarray(2);
+          daveSession.processWelcome(welcome);
+          if (daveSession.ready) evidence.mlsSessionReady = true;
+          log(`voice gw: processed op30 welcome (tid=${payload.readUInt16BE(0)}), sessionReady=${daveSession.ready}`);
+          evaluateGate();
+          break;
+        }
+        default:
+          log(`voice gw: unhandled DAVE binary op${op}`);
+      }
+    } catch (e) {
+      evidence.notes.push(`DAVE op${op} error: ${e.message}`);
+      log(`voice gw: DAVE op${op} handler error: ${e.message}`);
+    }
+  }
+
+  function daveKnownUsers() { return ['1513862586112671786']; }
+
+  function doUdpDiscoveryAndSelect(ready) {
+    udp = dgram.createSocket('udp4');
+    const disc = Buffer.alloc(74);
+    disc.writeUInt16BE(1, 0); disc.writeUInt16BE(70, 2); disc.writeUInt32BE(ready.ssrc, 4);
+    let selected = false;
+    udp.on('message', (msg) => {
+      if (selected) return; selected = true;
+      const ipEnd = msg.indexOf(0, 8);
+      const ip = msg.subarray(8, ipEnd).toString();
+      const port = msg.readUInt16BE(msg.length - 2);
+      const mode = (ready.modes || []).includes('aead_aes256_gcm_rtpsize') ? 'aead_aes256_gcm_rtpsize'
+        : (ready.modes || []).includes('aead_xchacha20_poly1305_rtpsize') ? 'aead_xchacha20_poly1305_rtpsize'
+        : (ready.modes || [])[0];
+      voiceState.mode = mode;
+      voiceWs.send(JSON.stringify({ op: 1, d: { protocol: 'udp', data: { address: ip, port, mode }, codecs: [
+        { name: 'opus', type: 'audio', priority: 1000, payload_type: 120 },
+        { name: 'VP8', type: 'video', priority: 1000, payload_type: 101, rtx_payload_type: 102 },
+        { name: 'H264', type: 'video', priority: 2000, payload_type: 103, rtx_payload_type: 104 },
+      ] }}));
+      log(`voice gw: sent SELECT PROTOCOL (mode=${mode}) after UDP discovery ${ip}:${port}`);
+    });
+    udp.on('error', (e) => log('udp error', e.message));
+    udp.send(disc, ready.port, ready.ip, (e) => { if (e) log('udp send err', e.message); else log('udp: sent IP discovery'); });
+  }
+
+  function evaluateGate() {
+    if (finished) return;
+    if (!evidence.voiceReady || evidence.negotiatedDaveVersion == null) return;
+    if (evidence.negotiatedDaveVersion === 0) {
+      // DAVE not enforced on this channel right now: gateway accepted, no E2EE membership needed.
+      return finish('PASS_NO_DAVE', 'voice gateway accepted; dave_protocol_version=0 (E2EE not active on this channel)');
+    }
+    if (evidence.mlsSessionReady) {
+      // Full E2EE membership achieved. Hold briefly to confirm the membership stays stable
+      // (no server disconnect), then leave cleanly.
+      log('gate: MLS session READY — holding 5s to confirm stable membership');
+      setTimeout(() => {
+        if (evidence.voiceCloseCode == null) evidence.notes.push('membership stable for 5s (no server disconnect)');
+        finish('PASS_MLS_READY', `DAVE v${evidence.negotiatedDaveVersion} negotiated; joined E2EE MLS group (session ready); privacyCode=${daveSession?.voicePrivacyCode || 'n/a'}`);
+      }, 5000);
+    }
+    // else: DAVE negotiated but MLS not yet complete — keep waiting for op25/27/29/30.
+  }
+}

dave/join.mjs

@@ -0,0 +1,241 @@
+// M1 — persistent selfbot voice join (builds on the proven gate.mjs handshake).
+//
+// Difference from gate.mjs: this does NOT leave after collecting evidence. It
+// joins the target voice channel, completes the DAVE/MLS E2EE membership, then
+// STAYS connected and reports:
+//   * who is speaking (op5 SPEAKING -> maps audio SSRC to a user id)
+//   * incoming UDP/RTP packets per SSRC (the media path the STT stage will read)
+//
+// This is the foundation for M2 (decrypt the incoming Opus and feed STT). No
+// audio is transmitted yet.
+//
+// Usage:
+//   node join.mjs                # join and stay until killed
+//   RUN_MS=15000 node join.mjs   # join, hold 15s, then leave (for verification)
+
+import WebSocket from 'ws';
+import dgram from 'node:dgram';
+import fs from 'node:fs';
+import * as davey from '@snazzah/davey';
+
+const env = Object.fromEntries(
+  fs.readFileSync(new URL('../.env', import.meta.url), 'utf8')
+    .split('\n').filter(l => l && !l.startsWith('#') && l.includes('='))
+    .map(l => { const i = l.indexOf('='); return [l.slice(0, i).trim(), l.slice(i + 1).trim()]; })
+);
+const TOKEN = env.DISCORD_SELFBOT_TOKEN;
+const GUILD_ID = process.env.GUILD_ID || '1352269198297923648';
+const CHANNEL_ID = process.env.CHANNEL_ID || '1352269198914621465';
+const SELF_ID = process.env.SELF_ID || '1513862586112671786';
+const DAVE_VER = process.env.DAVE_VER != null ? Number(process.env.DAVE_VER) : davey.DAVE_PROTOCOL_VERSION;
+const RUN_MS = process.env.RUN_MS != null ? Number(process.env.RUN_MS) : 0; // 0 = stay forever
+
+if (!TOKEN) { console.error('no DISCORD_SELFBOT_TOKEN in .env'); process.exit(2); }
+
+const t0 = Date.now();
+const log = (...a) => console.log(`[+${String(Date.now() - t0).padStart(6)}ms]`, ...a);
+console.log(`davey VERSION=${davey.VERSION} DAVE_PROTOCOL_VERSION=${davey.DAVE_PROTOCOL_VERSION}`);
+log(`join start: channel=${CHANNEL_ID} guild=${GUILD_ID} dave=${DAVE_VER}`);
+
+let mainWs, voiceWs, udp;
+let daveSession = null;
+let mlsReady = false;
+let discoverySelected = false;
+const knownUsers = new Set([SELF_ID]);
+const ssrcToUser = new Map();      // audio ssrc -> user id
+const rtpCount = new Map();        // ssrc -> packet count
+const voiceState = { session_id: null, token: null, endpoint: null, ssrc: null, ip: null, port: null, mode: null };
+
+let leaving = false;
+function leaveAndExit(code = 0) {
+  if (leaving) return;            // idempotent: hard ceiling + ready timer + signals must not double-fire
+  leaving = true;
+  try { mainWs?.send(JSON.stringify({ op: 4, d: { guild_id: GUILD_ID, channel_id: null, self_mute: true, self_deaf: true } })); } catch {}
+  setTimeout(() => {
+    try { voiceWs?.close(); } catch {}
+    try { udp?.close(); } catch {}
+    try { mainWs?.close(); } catch {}
+    log(`leaving. speakers seen: ${JSON.stringify([...ssrcToUser.entries()])}, rtp counts: ${JSON.stringify([...rtpCount.entries()])}`);
+    process.exit(code);
+  }, 400);
+}
+process.on('SIGINT', () => { log('SIGINT'); leaveAndExit(0); });
+process.on('SIGTERM', () => { log('SIGTERM'); leaveAndExit(0); });
+
+// Hard time-box ceiling, armed at startup regardless of handshake state. Without
+// this, a partial join (e.g. DAVE/MLS never completes op29/op30 so announceReady
+// never fires) would run the selfbot forever — a guardrail hole for a live test.
+if (RUN_MS > 0) setTimeout(() => { log(`RUN_MS=${RUN_MS} hard ceiling elapsed — leaving`); leaveAndExit(0); }, RUN_MS);
+
+// ---------- MAIN GATEWAY ----------
+mainWs = new WebSocket('wss://gateway.discord.gg/?v=10&encoding=json');
+let mainHb;
+mainWs.on('open', () => log('main gw: open'));
+mainWs.on('message', (raw) => {
+  const p = JSON.parse(raw.toString());
+  if (p.op === 10) {
+    mainHb = setInterval(() => { try { mainWs.send(JSON.stringify({ op: 1, d: null })); } catch {} }, p.d.heartbeat_interval);
+    mainWs.send(JSON.stringify({ op: 2, d: {
+      token: TOKEN,
+      capabilities: 16381,
+      properties: { os: 'Linux', browser: 'Chrome', device: '', system_locale: 'en-US', browser_user_agent: 'Mozilla/5.0', browser_version: '124.0', os_version: '', release_channel: 'stable', client_build_number: 300000 },
+      compress: false,
+      presence: { status: 'invisible', since: 0, activities: [], afk: false },
+    }}));
+    log('main gw: sent IDENTIFY');
+  } else if (p.op === 0) {
+    if (p.t === 'READY') {
+      log(`main gw: READY as ${p.d.user?.username} (${p.d.user?.id})`);
+      mainWs.send(JSON.stringify({ op: 4, d: { guild_id: GUILD_ID, channel_id: CHANNEL_ID, self_mute: false, self_deaf: false } }));
+      log('main gw: sent Voice State Update (join, mute=false + deaf=false so we can both speak and hear)');
+    } else if (p.t === 'VOICE_STATE_UPDATE' && p.d.user_id === SELF_ID && p.d.session_id) {
+      voiceState.session_id = p.d.session_id; log('VOICE_STATE_UPDATE session_id acquired'); maybeConnectVoice();
+    } else if (p.t === 'VOICE_SERVER_UPDATE') {
+      voiceState.token = p.d.token; voiceState.endpoint = p.d.endpoint;
+      log(`VOICE_SERVER_UPDATE endpoint=${p.d.endpoint}`); maybeConnectVoice();
+    }
+  }
+});
+mainWs.on('close', (c, r) => { log(`main gw: close ${c} ${r}`); clearInterval(mainHb); });
+mainWs.on('error', (e) => log('main gw error', e.message));
+
+// ---------- VOICE GATEWAY ----------
+function maybeConnectVoice() {
+  if (voiceWs || !voiceState.session_id || !voiceState.token || !voiceState.endpoint) return;
+  const url = `wss://${voiceState.endpoint}/?v=8`;
+  log(`voice gw: connecting ${url}`);
+  voiceWs = new WebSocket(url);
+  let voiceHb, lastSeq = null;
+
+  voiceWs.on('open', () => {
+    voiceWs.send(JSON.stringify({ op: 0, d: {
+      server_id: GUILD_ID, user_id: SELF_ID, session_id: voiceState.session_id, token: voiceState.token,
+      max_dave_protocol_version: DAVE_VER,
+    }}));
+    log(`voice gw: sent IDENTIFY (max_dave_protocol_version=${DAVE_VER})`);
+  });
+  voiceWs.on('message', (raw, isBinary) => {
+    if (isBinary) {
+      const buf = Buffer.isBuffer(raw) ? raw : Buffer.from(raw);
+      const seq = buf.readUInt16BE(0), op = buf.readUInt8(2), payload = buf.subarray(3);
+      lastSeq = seq; onDaveBinary(op, payload, seq);
+      return;
+    }
+    handleVoiceJson(JSON.parse(raw.toString()));
+  });
+  voiceWs.on('close', (c, r) => {
+    log(`voice gw: CLOSE code=${c} reason="${r?.toString() || ''}"`);
+    clearInterval(voiceHb);
+    if (c === 4017) { log('FATAL: close 4017 (DAVE rejected)'); leaveAndExit(1); }
+  });
+  voiceWs.on('error', (e) => log('voice gw error', e.message));
+
+  function handleVoiceJson(p) {
+    switch (p.op) {
+      case 8:
+        voiceHb = setInterval(() => { try { voiceWs.send(JSON.stringify({ op: 3, d: { t: Date.now(), seq_ack: lastSeq ?? 0 } })); } catch {} }, p.d.heartbeat_interval);
+        log(`voice gw: HELLO hb=${p.d.heartbeat_interval}`);
+        break;
+      case 2: // READY
+        voiceState.ssrc = p.d.ssrc; voiceState.ip = p.d.ip; voiceState.port = p.d.port;
+        log(`voice gw: READY ssrc=${p.d.ssrc} udp=${p.d.ip}:${p.d.port} modes=${JSON.stringify(p.d.modes)}`);
+        doUdpDiscoveryAndSelect(p.d);
+        break;
+      case 4: // SESSION_DESCRIPTION
+        voiceState.daveVer = p.d?.dave_protocol_version ?? null;
+        log(`voice gw: SESSION_DESCRIPTION dave=${voiceState.daveVer} mode=${p.d?.mode}`);
+        announceReady();
+        break;
+      case 5: // SPEAKING — maps a user to their audio ssrc
+        if (p.d?.user_id && p.d?.ssrc != null) {
+          ssrcToUser.set(p.d.ssrc, p.d.user_id);
+          knownUsers.add(p.d.user_id);
+          log(`voice gw: SPEAKING user=${p.d.user_id} ssrc=${p.d.ssrc} flags=${p.d.speaking}`);
+        }
+        break;
+      case 11: for (const u of (p.d.user_ids || [])) knownUsers.add(u); log(`voice gw: op11 clients_connect ${JSON.stringify(p.d.user_ids)}`); break;
+      case 13: if (p.d?.user_id) { knownUsers.delete(p.d.user_id); log(`voice gw: op13 client_disconnect ${p.d.user_id}`); } break;
+      case 20: if (p.d.user_id) knownUsers.add(p.d.user_id); break;
+      case 21: try { voiceWs.send(JSON.stringify({ op: 23, d: { transition_id: p.d.transition_id } })); } catch {} log('voice gw: DAVE op21 prepare_transition (acked)'); break;
+      case 22: log('voice gw: DAVE op22 execute_transition'); break;
+      default: /* quiet */ break;
+    }
+  }
+
+  function onDaveBinary(op, payload) {
+    try {
+      switch (op) {
+        case 25: {
+          if (!daveSession) daveSession = new davey.DAVESession(voiceState.daveVer || DAVE_VER, SELF_ID, CHANNEL_ID);
+          daveSession.setExternalSender(payload);
+          const kp = daveSession.getSerializedKeyPackage();
+          voiceWs.send(Buffer.concat([Buffer.from([26]), kp]), { binary: true });
+          log(`voice gw: sent DAVE op26 key_package len=${kp.length}`);
+          break;
+        }
+        case 27: {
+          const res = daveSession.processProposals(payload.readUInt8(0), payload.subarray(1), Array.from(knownUsers));
+          if (res && res.commit) {
+            const parts = [Buffer.from([28]), res.commit];
+            if (res.welcome) parts.push(res.welcome);
+            voiceWs.send(Buffer.concat(parts), { binary: true });
+            log(`voice gw: sent DAVE op28 commit_welcome`);
+          }
+          break;
+        }
+        case 29: daveSession.processCommit(payload.subarray(2)); mlsReady = daveSession.ready; log(`voice gw: op29 commit -> ready=${mlsReady}`); announceReady(); break;
+        case 30: daveSession.processWelcome(payload.subarray(2)); mlsReady = daveSession.ready; log(`voice gw: op30 welcome -> ready=${mlsReady}`); announceReady(); break;
+        default: break;
+      }
+    } catch (e) { log(`voice gw: DAVE op${op} error: ${e.message}`); }
+  }
+
+  let announced = false;
+  function announceReady() {
+    if (announced) return;
+    const daveOk = voiceState.daveVer === 0 || mlsReady;
+    if (voiceState.ssrc != null && voiceState.daveVer != null && daveOk && discoverySelected) {
+      announced = true;
+      log(`✅ JOINED & READY. channel=${CHANNEL_ID} dave=${voiceState.daveVer} mlsReady=${mlsReady} privacyCode=${daveSession?.voicePrivacyCode || 'n/a'}`);
+      log('   staying connected, listening for speakers…');
+      // time-box is owned by the startup hard-ceiling timer (armed regardless of ready state)
+    }
+  }
+
+  function doUdpDiscoveryAndSelect(ready) {
+    udp = dgram.createSocket('udp4');
+    const disc = Buffer.alloc(74);
+    disc.writeUInt16BE(1, 0); disc.writeUInt16BE(70, 2); disc.writeUInt32BE(ready.ssrc, 4);
+    udp.on('message', (msg) => {
+      if (!discoverySelected) {
+        discoverySelected = true;
+        const ipEnd = msg.indexOf(0, 8);
+        const ip = msg.subarray(8, ipEnd).toString();
+        const port = msg.readUInt16BE(msg.length - 2);
+        const mode = (ready.modes || []).includes('aead_aes256_gcm_rtpsize') ? 'aead_aes256_gcm_rtpsize'
+          : (ready.modes || []).includes('aead_xchacha20_poly1305_rtpsize') ? 'aead_xchacha20_poly1305_rtpsize'
+          : (ready.modes || [])[0];
+        voiceState.mode = mode;
+        voiceWs.send(JSON.stringify({ op: 1, d: { protocol: 'udp', data: { address: ip, port, mode }, codecs: [
+          { name: 'opus', type: 'audio', priority: 1000, payload_type: 120 },
+        ] }}));
+        log(`voice gw: sent SELECT PROTOCOL (mode=${mode}) after UDP discovery ${ip}:${port}`);
+        announceReady();
+        return;
+      }
+      // After selection: these are incoming SRTP media packets. Just tally per-SSRC
+      // (decryption -> Opus -> PCM is M2). RTP: ssrc at bytes 8..11, pt = byte1 & 0x7f.
+      if (msg.length >= 12) {
+        const ssrc = msg.readUInt32BE(8);
+        rtpCount.set(ssrc, (rtpCount.get(ssrc) || 0) + 1);
+        const n = rtpCount.get(ssrc);
+        if (n === 1 || n % 200 === 0) {
+          const user = ssrcToUser.get(ssrc) || '?';
+          log(`rtp: ssrc=${ssrc} user=${user} pt=${msg.readUInt8(1) & 0x7f} count=${n}`);
+        }
+      }
+    });
+    udp.on('error', (e) => log('udp error', e.message));
+    udp.send(disc, ready.port, ready.ip, (e) => { if (e) log('udp send err', e.message); else log('udp: sent IP discovery'); });
+  }
+}

dave/package.json

@@ -0,0 +1,16 @@
+{
+  "name": "wsai-dave-poc",
+  "private": true,
+  "type": "module",
+  "description": "Discord voice layer for watch_sceen_ai: official bot joins target voice channel (DAVE/MLS E2EE) and receives per-user Opus audio for STT. bot.mjs = current path; gate.mjs/join.mjs = legacy selfbot (kept for deferred video track).",
+  "main": "bot.mjs",
+  "dependencies": {
+    "@discordjs/opus": "^0.10.0",
+    "@discordjs/voice": "^0.19.2",
+    "@snazzah/davey": "^0.1.12",
+    "discord.js": "^14.27.0",
+    "libsodium-wrappers": "^0.8.4",
+    "prism-media": "^1.3.5",
+    "ws": "^8.18.0"
+  }
+}

docker-entrypoint.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Test/run dispatcher for the watch_sceen_ai container.
+#
+#   smoke  (default)  pytest smoke suite (mock pipeline, no deps, no GPU)
+#   voice             python -m wsai --voice  (eyes-free mock voice loop demo)
+#   mock              python -m wsai          (full mock flow, few frames)
+#   gpu               nvidia-smi — prove the GPU is visible inside the container
+#   join              node dave/bot.mjs — LIVE official-bot voice join (needs .env
+#                     mounted; honors RUN_MS / GUILD_ID / CHANNEL_ID)
+#   shell             drop to bash
+set -euo pipefail
+cd /app
+
+cmd="${1:-smoke}"; shift || true
+
+case "$cmd" in
+  smoke)  exec python -m pytest -q ;;
+  voice)  exec python -m wsai --voice ;;
+  mock)   exec python -m wsai ;;
+  gpu)    exec nvidia-smi ;;
+  join)   cd dave && exec node bot.mjs "$@" ;;
+  shell)  exec bash ;;
+  *)      exec "$cmd" "$@" ;;
+esac

poc/capture_xvfb.py

@@ -0,0 +1,75 @@
+"""PoC: prove that a non-headless Chromium under a virtual display (Xvfb) can
+render a live/animated page and that we can capture it as frames.
+
+This de-risks the hardest unknown of decision #1 (watch the real Discord screen
+share by capturing a real client's rendered pixels) BEFORE touching Discord.
+
+Run under a virtual display:
+
+    xvfb-run -a --server-args="-screen 0 1280x720x24" \
+        .venv/bin/python poc/capture_xvfb.py
+
+Success = frames are non-blank AND change over time (proving live rendering +
+capture actually work on this headless server).
+"""
+
+from __future__ import annotations
+
+import hashlib
+import pathlib
+import sys
+import time
+
+from playwright.sync_api import sync_playwright
+
+HERE = pathlib.Path(__file__).parent
+PAGE = (HERE / "test_page.html").resolve()
+OUT = HERE / "frames"
+N = 6
+INTERVAL = 0.4
+
+
+def main() -> int:
+    OUT.mkdir(exist_ok=True)
+    hashes: list[str] = []
+    sizes: list[int] = []
+
+    with sync_playwright() as p:
+        browser = p.chromium.launch(
+            channel="chrome",  # use system google-chrome, no browser download
+            headless=False,    # WebRTC/video needs a real (virtual) display
+            args=["--no-sandbox", "--disable-gpu", "--autoplay-policy=no-user-gesture-required"],
+        )
+        page = browser.new_page(viewport={"width": 1280, "height": 720})
+        page.goto(PAGE.as_uri())
+        page.wait_for_timeout(500)
+
+        for i in range(N):
+            png = page.screenshot()  # captures the rendered virtual screen
+            f = OUT / f"frame_{i:02d}.png"
+            f.write_bytes(png)
+            h = hashlib.sha256(png).hexdigest()[:12]
+            hashes.append(h)
+            sizes.append(len(png))
+            print(f"  frame {i}: {len(png):>7} bytes  sha={h}")
+            time.sleep(INTERVAL)
+
+        browser.close()
+
+    distinct = len(set(hashes))
+    min_size = min(sizes)
+    print(f"\n{N} frames, {distinct} distinct, min size {min_size} bytes")
+
+    # A blank/black 1280x720 PNG compresses to a few KB; a real rendered scene is
+    # much larger. Require non-trivial size and that the scene actually moved.
+    ok_nonblank = min_size > 8_000
+    ok_changing = distinct >= 3
+    if ok_nonblank and ok_changing:
+        print("PoC PASS: Xvfb + Chromium rendered a live scene and we captured changing, non-blank frames.")
+        return 0
+    print(f"PoC FAIL: nonblank={ok_nonblank} changing={ok_changing}")
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

poc/test_page.html

@@ -0,0 +1,29 @@
+<!doctype html>
+<html>
+<head><meta charset="utf-8"><title>capture test</title>
+<style>html,body{margin:0;background:#101418}</style></head>
+<body>
+<canvas id="c" width="1280" height="720"></canvas>
+<script>
+// An always-changing scene, so captured frames must differ frame-to-frame and
+// be non-blank. This stands in for "a Discord screen share is playing".
+const c = document.getElementById('c'), x = c.getContext('2d');
+let t = 0;
+function draw() {
+  t++;
+  x.fillStyle = '#101418'; x.fillRect(0, 0, 1280, 720);
+  // moving box
+  const px = (t * 7) % 1180;
+  x.fillStyle = `hsl(${(t*3)%360} 80% 55%)`;
+  x.fillRect(px, 300 + 120 * Math.sin(t / 15), 100, 100);
+  // big live counter text
+  x.fillStyle = '#e8eef2'; x.font = 'bold 90px monospace';
+  x.fillText('FRAME ' + t, 60, 140);
+  x.font = '28px monospace';
+  x.fillText(new Date().toISOString(), 60, 200);
+  requestAnimationFrame(draw);
+}
+draw();
+</script>
+</body>
+</html>

requirements.txt

@@ -0,0 +1,20 @@
+# Core skeleton has NO required third-party deps (mock mode is pure stdlib).
+# Install extras per backend you enable:
+
+# --- screen capture (WSAI_SOURCE=mss) ---
+# mss
+# pillow
+
+# --- cloud eyes + brain (WSAI_VISION=claude / WSAI_BRAIN=claude) ---
+# anthropic
+
+# --- voice backends (run in their OWN venvs; loaded as persistent workers) ---
+# STT: faster-whisper  (WSAI_STT=whisper) — installed in a dedicated venv, e.g.
+#   uv venv --python 3.12 /home/claude/jarvis-stt/whisper312
+#   uv pip install --python /home/claude/jarvis-stt/whisper312/bin/python faster-whisper
+#   (override interpreter/model via WSAI_WHISPER_PYTHON / WSAI_WHISPER_MODEL)
+# TTS: MeloTTS         (WSAI_TTS=melo)     — in /home/claude/jarvis-tts/melo311
+# (Opus decode for Discord voice audio, e.g. via the selfbot/ffmpeg path — pending)
+
+# --- dev ---
+# pytest

tests/test_emotion.py

@@ -0,0 +1,90 @@
+"""Emotion-tag parsing for expressive TTS. A ``[감정]`` tag must steer the
+pitch/speed of the text that follows without being spoken; a bracket that is NOT
+a known emotion word must be kept as ordinary spoken content."""
+
+from wsai.backends.emotion import (
+    EMOTION_PARAMS,
+    match_emotion,
+    parse_segments,
+)
+from wsai.dashboard import _speech_text
+
+BASE = 1.3
+
+
+def test_emotion_tag_is_not_spoken_and_sets_delivery():
+    segs = parse_segments("[기쁨] 오늘 날씨 좋다", BASE)
+    assert len(segs) == 1
+    assert "기쁨" not in segs[0].text        # the tag word is dropped
+    assert segs[0].text == "오늘 날씨 좋다"
+    mult, semis = EMOTION_PARAMS["happy"]
+    assert segs[0].speed == BASE * mult
+    assert segs[0].pitch == semis
+
+
+def test_midreply_emotion_change_splits_segments():
+    segs = parse_segments("[속상함] 정말 힘들었겠다. [힘차게] 하지만 넌 할 수 있어!", BASE)
+    assert len(segs) == 2
+    assert segs[0].text == "정말 힘들었겠다."
+    assert segs[1].text == "하지만 넌 할 수 있어!"
+    assert segs[0].pitch == EMOTION_PARAMS["sad"][1]
+    assert segs[1].pitch == EMOTION_PARAMS["hopeful"][1]
+
+
+def test_non_emotion_bracket_is_spoken_without_brackets():
+    segs = parse_segments("[기쁨] 첫째는 [1번] 항목이야", BASE)
+    assert len(segs) == 1
+    # "1번" is not an emotion -> read it; brackets themselves are gone.
+    assert "1번" in segs[0].text
+    assert "[" not in segs[0].text and "]" not in segs[0].text
+    assert segs[0].pitch == EMOTION_PARAMS["happy"][1]
+
+
+def test_text_before_first_tag_is_neutral():
+    segs = parse_segments("잠깐만. [신남] 찾았다!", BASE)
+    assert segs[0].text == "잠깐만."
+    assert segs[0].speed == BASE and segs[0].pitch == 0.0
+    assert segs[1].pitch == EMOTION_PARAMS["excited"][1]
+
+
+def test_plain_text_is_one_neutral_segment():
+    segs = parse_segments("그냥 평범한 문장이야", BASE)
+    assert len(segs) == 1
+    assert segs[0].speed == BASE and segs[0].pitch == 0.0
+
+
+def test_empty_input_yields_no_segments():
+    assert parse_segments("", BASE) == []
+    assert parse_segments("   ", BASE) == []
+
+
+def test_only_emotion_tags_yield_no_segments():
+    # A reply that is nothing but tags has nothing to say.
+    assert parse_segments("[기쁨][신남]", BASE) == []
+
+
+def test_match_emotion_is_synonym_and_space_tolerant():
+    assert match_emotion("속상함") == "sad"
+    assert match_emotion(" 힘 차게 ") == "hopeful"   # squeezed + trimmed
+    assert match_emotion("행복하게") == "happy"
+    assert match_emotion("메모") is None            # not an emotion
+
+
+def test_persona_examples_are_all_recognised():
+    # Every emotion the brain persona advertises must resolve, or it would be
+    # read aloud instead of shaping the voice.
+    for word in ["힘차게", "궁금", "반가움", "차분하게", "웃으며", "속상함"]:
+        assert match_emotion(word) is not None, word
+
+
+def test_voice_turn_keeps_leading_emotion_tag_for_tts_parser():
+    text = "[속상함] 정말 힘들었겠다. [힘차게] 하지만 넌 할 수 있어!"
+
+    spoken = _speech_text(text)
+    segs = parse_segments(spoken, BASE)
+
+    assert spoken == text
+    assert segs[0].text == "정말 힘들었겠다."
+    assert segs[0].pitch == EMOTION_PARAMS["sad"][1]
+    assert segs[1].text == "하지만 넌 할 수 있어!"
+    assert segs[1].pitch == EMOTION_PARAMS["hopeful"][1]

tests/test_monitor.py

@@ -0,0 +1,119 @@
+"""The monitor must record step-by-step turns (heard / thought / answered,
+per-step timing, ok vs error) and stream them to subscribers — that data is
+exactly what the status dashboard renders."""
+
+import asyncio
+import json
+
+from wsai.backends.mock import MockBrain, MockSTT, MockTTS
+from wsai.monitor import Monitor
+from wsai.pipeline import Pipeline
+
+
+def test_monitor_records_turn_with_timed_steps():
+    async def go():
+        mon = Monitor()
+
+        pipe = Pipeline(
+            brain=MockBrain(),
+            stt=MockSTT(script=["안녕"], interval=0.01),
+            tts=MockTTS(),
+            monitor=mon,
+        )
+        await asyncio.wait_for(pipe.run(), timeout=5)
+        return mon
+
+    mon = asyncio.run(go())
+    snap = mon.snapshot()
+
+    assert snap["status"]["turns_total"] == 1
+    assert snap["status"]["running"] is False  # cleaned up after run
+    assert len(snap["turns"]) == 1
+
+    turn = snap["turns"][0]
+    assert turn["heard"] == "안녕"          # what it heard
+    assert turn["reply"]                     # what it answered
+    assert turn["status"] == "ok"            # it worked
+    assert turn["total_ms"] >= 0
+    # step-by-step: every stage is named and timed
+    names = [s["name"] for s in turn["steps"]]
+    assert names == ["화면 맥락", "두뇌(생각)", "응답(TTS/전송)"]
+    assert all(s["ok"] is True for s in turn["steps"])
+    assert all(s["ms"] >= 0 for s in turn["steps"])
+
+
+def test_monitor_marks_errors():
+    class BoomBrain(MockBrain):
+        async def respond(self, user_text, screen, history):
+            raise RuntimeError("boom")
+
+    async def go():
+        mon = Monitor()
+        pipe = Pipeline(
+            brain=BoomBrain(),
+            stt=MockSTT(script=["안녕"], interval=0.01),
+            tts=MockTTS(),
+            monitor=mon,
+        )
+        try:
+            await asyncio.wait_for(pipe.run(), timeout=5)
+        except BaseException:
+            pass  # TaskGroup re-raises; we only care about recorded telemetry
+        return mon
+
+    mon = asyncio.run(go())
+    snap = mon.snapshot()
+
+    turn = snap["turns"][0]
+    assert turn["status"] == "error"
+    brain_step = next(s for s in turn["steps"] if s["name"] == "두뇌(생각)")
+    assert brain_step["ok"] is False
+    assert "boom" in brain_step["error"]
+    assert snap["status"]["errors_total"] >= 1
+
+
+def test_error_turn_increments_errors_total_once():
+    """The dashboard voice turn (dashboard.voice_turn) records a turn and calls
+    turn.finish(error=...) directly — it never goes through the pipeline's
+    log("error") path. That failure must still land in errors_total, and exactly
+    once no matter how many times the turn is re-published."""
+    mon = Monitor()
+    turn = mon.turn(source="discord")
+    turn.heard("`코드` 얘기")           # touches/publishes again
+    turn.replied("답변")                 # and again
+    assert mon.status_snapshot()["errors_total"] == 0
+
+    turn.finish(error="melo synth failed: KeyError '`'")
+    assert mon.status_snapshot()["errors_total"] == 1
+
+    # A stray re-finish/re-publish must not double count.
+    turn.finish(error="melo synth failed: KeyError '`'")
+    turn._touch()
+    assert mon.status_snapshot()["errors_total"] == 1
+
+    # The failure is also visible as an error-level event for the live feed.
+    events = mon.snapshot()["events"]
+    assert any(e["type"] == "log" and e["level"] == "error" for e in events)
+
+
+def test_subscriber_receives_live_turn_events():
+    async def go():
+        mon = Monitor()
+        q = mon.subscribe()
+        pipe = Pipeline(
+            brain=MockBrain(),
+            stt=MockSTT(script=["안녕"], interval=0.01),
+            tts=MockTTS(),
+            monitor=mon,
+        )
+        await asyncio.wait_for(pipe.run(), timeout=5)
+        return q
+
+    q = asyncio.run(go())
+    events = []
+    while not q.empty():
+        events.append(json.loads(q.get_nowait()))
+
+    types = {e["type"] for e in events}
+    assert "turn" in types   # live turn updates were pushed
+    assert "status" in types  # listening/running status changes were pushed

tests/test_pipeline.py

@@ -0,0 +1,136 @@
+"""Smoke test: the mock pipeline must run end-to-end and route screen context
+into the brain's replies."""
+
+import asyncio
+
+import pytest
+
+from wsai.backends.mock import (
+    MockBrain,
+    MockFrameSource,
+    MockSTT,
+    MockTTS,
+    MockVision,
+)
+from wsai.interfaces import Frame
+from wsai.pipeline import Pipeline
+
+
+def test_mock_pipeline_runs_and_replies(capsys):
+    replies: list[str] = []
+
+    class CapturingTTS(MockTTS):
+        async def speak(self, reply):
+            replies.append(reply.text)
+
+    pipe = Pipeline(
+        source=MockFrameSource(interval=0.05, limit=3),
+        vision=MockVision(),
+        brain=MockBrain(),
+        stt=MockSTT(script=["화면에 뭐 보여?"], interval=0.1),
+        tts=CapturingTTS(),
+    )
+
+    asyncio.run(asyncio.wait_for(pipe.run(), timeout=5))
+
+    assert replies, "brain produced no reply"
+    # The reply must embed the screen observation → context reached the brain.
+    assert "화면:" in replies[0]
+
+
+def test_voice_only_pipeline_runs_without_eyes():
+    """Eyes-free config (no source/vision) still runs STT -> Brain -> TTS."""
+    replies: list[str] = []
+
+    class CapturingTTS(MockTTS):
+        async def speak(self, reply):
+            replies.append(reply.text)
+
+    pipe = Pipeline(
+        brain=MockBrain(),
+        stt=MockSTT(script=["안녕", "잘 있어"], interval=0.05),
+        tts=CapturingTTS(),
+    )
+
+    asyncio.run(asyncio.wait_for(pipe.run(), timeout=5))
+
+    assert len(replies) == 2, "voice loop did not reply to every utterance"
+    # No eyes → the brain must report it has not seen a screen.
+    assert "아직 화면을 못 읽었어요" in replies[0]
+
+
+def test_error_in_one_loop_cancels_siblings_and_closes():
+    """If the conversation loop raises, the perception loop must be cancelled
+    (not left running detached) and every source must still be closed — i.e. no
+    close-during-use and no orphaned task."""
+    closed = {"source": False, "stt": False}
+
+    class ForeverSource:
+        async def frames(self):
+            while True:
+                await asyncio.sleep(0.01)
+                yield Frame(data=b"", width=1, height=1, ts=0.0)
+
+        async def aclose(self):
+            closed["source"] = True
+
+    class BoomSTT(MockSTT):
+        async def aclose(self):
+            closed["stt"] = True
+
+    class BoomBrain(MockBrain):
+        async def respond(self, user_text, screen, history):
+            raise RuntimeError("boom")
+
+    pipe = Pipeline(
+        source=ForeverSource(),
+        vision=MockVision(),
+        brain=BoomBrain(),
+        stt=BoomSTT(script=["hi"], interval=0.01),
+        tts=MockTTS(),
+    )
+
+    with pytest.raises(BaseException):  # TaskGroup raises an ExceptionGroup
+        asyncio.run(asyncio.wait_for(pipe.run(), timeout=5))
+
+    assert closed["source"] is True, "perception source was not closed (orphaned loop)"
+    assert closed["stt"] is True, "stt was not closed"
+
+
+def test_history_is_bounded():
+    pipe = Pipeline(
+        source=MockFrameSource(limit=0),
+        vision=MockVision(),
+        brain=MockBrain(),
+        history_turns=3,
+    )
+    for i in range(10):
+        pipe._remember(f"u{i}", f"a{i}")
+    assert len(pipe._history) == 3
+    assert pipe._history[-1] == ("u9", "a9")
+
+
+def test_prewarm_warms_backends_and_survives_failure():
+    """Backends exposing warmup() are preloaded at startup; a warmup that
+    raises is logged but must not abort the run (first-utterance latency is an
+    optimization, not a hard requirement)."""
+    warmed: list[str] = []
+
+    class WarmTTS(MockTTS):
+        async def warmup(self):
+            warmed.append("tts")
+
+    class BoomWarmSTT(MockSTT):
+        async def warmup(self):
+            warmed.append("stt")
+            raise RuntimeError("model unavailable")
+
+    pipe = Pipeline(
+        brain=MockBrain(),
+        stt=BoomWarmSTT(script=["안녕"], interval=0.01),
+        tts=WarmTTS(),
+    )
+
+    asyncio.run(asyncio.wait_for(pipe.run(), timeout=5))
+
+    assert "tts" in warmed and "stt" in warmed, "warmup() not called on backends"

tests/test_textnorm.py

@@ -0,0 +1,39 @@
+"""TTS input normalisation. Claude speaks in markdown/backticks; MeloTTS's
+Korean normaliser crashes on a bare backtick (``KeyError: '`'``), which used to
+take down the whole voice turn. normalize_for_speech() must strip formatting and
+above all guarantee no backtick reaches the synthesiser."""
+
+from wsai.backends.melo import normalize_for_speech
+
+
+def test_backticks_are_always_removed():
+    # The exact crash trigger: inline code, a fenced block, and a stray backtick.
+    reply = "`ls -la` 를 써봐. 예시:\n```python\nprint('hi')\n```\n그리고 ` 이건 홀로 남은 백틱"
+    out = normalize_for_speech(reply)
+    assert "`" not in out            # the character that crashes MeloTTS is gone
+    assert "ls -la" in out           # inner words are kept, just unwrapped
+    assert "print('hi')" in out      # fenced code content survives as spoken text
+
+
+def test_markdown_structure_flattened():
+    reply = "# 제목\n- 첫째 항목\n- 둘째 항목\n**굵게** 그리고 _기울임_\n> 인용문"
+    out = normalize_for_speech(reply)
+    assert "#" not in out
+    assert "**" not in out and "_" not in out
+    assert not out.lstrip().startswith(("-", ">"))
+    assert "첫째 항목" in out and "굵게" in out and "인용문" in out
+
+
+def test_links_reduced_to_label():
+    out = normalize_for_speech("자세히는 [문서](https://example.com/docs) 참고해")
+    assert "문서" in out
+    assert "http" not in out and "]" not in out and "(" not in out
+
+
+def test_plain_text_is_left_intact():
+    plain = "안녕, 지금 화면 잘 보고 있어. 뭐 도와줄까?"
+    assert normalize_for_speech(plain) == plain
+
+
+def test_empty_is_safe():
+    assert normalize_for_speech("") == ""

tests/test_whisper_stt.py

@@ -0,0 +1,60 @@
+"""Unit tests for the WhisperSTT source plumbing.
+
+These do NOT load a model or spawn the worker (that needs the whisper312 venv and
+is exercised by the manual TTS->STT round trip). They pin the SpeechToText
+contract: how `utterances()` turns an audio source into Utterances, and how it
+behaves with no source wired yet.
+"""
+
+import asyncio
+from typing import AsyncIterator
+
+from wsai.backends.whisper import WhisperSTT
+from wsai.interfaces import SpeechToText, Utterance
+
+
+def _collect(stt: WhisperSTT) -> list[Utterance]:
+    async def run():
+        return [u async for u in stt.utterances()]
+
+    return asyncio.run(asyncio.wait_for(run(), timeout=5))
+
+
+async def _paths(items) -> AsyncIterator[str]:
+    for it in items:
+        yield it
+
+
+def test_is_speech_to_text():
+    assert isinstance(WhisperSTT(), SpeechToText)
+
+
+def test_no_audio_source_yields_nothing():
+    # Discord voice receiver not wired yet -> the loop idles and returns.
+    assert _collect(WhisperSTT(audio_source=None)) == []
+
+
+def test_utterances_transcribes_each_chunk(monkeypatch):
+    stt = WhisperSTT(audio_source=_paths(["a.wav", "b.wav"]))
+
+    async def fake_transcribe(wav_path, *, language=None):
+        return f"text::{wav_path}"
+
+    monkeypatch.setattr(stt, "transcribe", fake_transcribe)
+
+    utts = _collect(stt)
+    assert [u.text for u in utts] == ["text::a.wav", "text::b.wav"]
+    assert all(u.source == "voice" for u in utts)
+
+
+def test_empty_transcript_is_skipped(monkeypatch):
+    # Silence / VAD-filtered audio transcribes to "" and must not become a turn.
+    stt = WhisperSTT(audio_source=_paths(["silent.wav", "real.wav"]))
+
+    async def fake_transcribe(wav_path, *, language=None):
+        return "" if wav_path == "silent.wav" else "안녕"
+
+    monkeypatch.setattr(stt, "transcribe", fake_transcribe)
+
+    utts = _collect(stt)
+    assert [u.text for u in utts] == ["안녕"]

wsai/__init__.py

@@ -0,0 +1,8 @@
+"""watch_screen_ai — an AI that watches a shared screen and talks with you."""
+
+from .config import Settings
+from .factory import build
+from .pipeline import Pipeline
+
+__all__ = ["Settings", "build", "Pipeline"]
+__version__ = "0.0.1"

wsai/__main__.py

@@ -0,0 +1,232 @@
+"""Entry point.
+
+    python -m wsai              # mock pipeline (no deps, no keys) — runs a demo
+    python -m wsai --voice      # eyes-free voice loop demo (STT -> Brain -> TTS)
+    python -m wsai --dashboard  # live status website + a short voice demo, then idle
+    python -m wsai --live       # capture this screen + Claude eyes/brain
+    python -m wsai --env        # build from WSAI_* environment variables
+
+The mock run is bounded (a few frames + a scripted conversation) so it exits on
+its own; --dashboard keeps the site open after a short demo; --live/--env run
+until Ctrl-C.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import logging
+import os
+import socket
+
+from .config import Settings
+from .factory import build
+from .monitor import Monitor
+
+
+def _lan_ip() -> str:
+    try:
+        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        s.connect(("8.8.8.8", 80))
+        ip = s.getsockname()[0]
+        s.close()
+        return ip
+    except OSError:
+        return "127.0.0.1"
+
+
+async def _run(
+    settings: Settings,
+    demo: bool,
+    monitor: Monitor | None,
+    *,
+    demo_loop: bool = False,
+    keep_dashboard_open: bool = False,
+) -> None:
+    if demo:
+        # Bounded demo so CI / a quick check terminates.
+        from .backends.mock import MockFrameSource, MockSTT
+
+        pipe = build(settings, monitor=monitor)
+        if pipe.source is not None:  # keep eyes-free configs eyes-free
+            pipe.source = MockFrameSource(interval=0.3, limit=4)
+        if pipe.stt is not None:
+            pipe.stt = MockSTT(interval=2.0 if demo_loop else 0.4, loop=demo_loop)
+        await pipe.run()
+        if keep_dashboard_open:
+            # Keep the status page alive without generating fake conversations
+            # forever. The previous default looped mock STT indefinitely, which
+            # made the dashboard look like it had heard thousands of real users.
+            if monitor is not None:
+                monitor.set_status(running=True, listening=False)
+                monitor.log("info", "mock 데모 완료 — 실제 음성 파이프라인 연결 대기")
+            await asyncio.Event().wait()
+        return
+    await build(settings, monitor=monitor).run()
+
+
+def _run_stt_test(host: str, port: int) -> None:
+    """Serve the dashboard with a real GPU STT backend so a human can test
+    recognition from the browser (record mic or upload an audio file). No mock
+    conversation loop — the page just hosts the recognition test."""
+    import time
+
+    from .backends.whisper import WhisperSTT
+    from .dashboard import Dashboard
+    from .monitor import Monitor
+
+    monitor = Monitor()
+    stt = WhisperSTT()
+    dash = Dashboard(monitor, host=host, port=port, stt=stt)
+    dash.start()
+    monitor.set_components({"source": "none", "vision": "none", "stt": "whisper",
+                            "brain": "none", "tts": "none"})
+    monitor.set_status(running=True, listening=False)
+    monitor.log("info", "STT 인식 테스트 서버 시작 — GPU 워밍업 중…")
+    print("\n  STT 워밍업 중… (모델 로드 + CUDA 예열)")
+    dash.warm()  # load + warm the GPU worker so the first recognition is instant
+    dev = getattr(stt, "resolved_device", None) or "?"
+    monitor.log("info", f"STT 준비 완료 (device={dev}). 녹음/파일 업로드로 인식하세요.")
+
+    shown = host if host not in ("0.0.0.0", "") else _lan_ip()
+    print(f"\n  음성 인식 테스트 사이트:  http://{shown}:{port}   (STT device: {dev})")
+    print(f"  (로컬:                  http://127.0.0.1:{port} )\n")
+    try:
+        while True:
+            time.sleep(3600)
+    except KeyboardInterrupt:
+        pass
+    finally:
+        dash.stop()
+
+
+def _run_voice_server(host: str, port: int) -> None:
+    """Serve the STT+TTS voice-turn endpoint that the Discord bot (dave/bot.mjs)
+    calls: it POSTs a captured utterance wav and gets back the reply wav to play
+    into the voice channel. Both STT and TTS run on the GPU and are pre-warmed.
+    The same page also shows the live turn feed. Echo mode for now (the reply is
+    what was heard); the Claude brain can be added as the next slice."""
+    import time
+
+    from .backends.melo import MeloTTS
+    from .backends.whisper import WhisperSTT
+    from .dashboard import Dashboard
+    from .monitor import Monitor
+
+    # Real Claude brain (think + reply). If it can't be constructed (no anthropic
+    # package / no Claude auth), fall back to echo so the loop still works.
+    brain = None
+    brain_name = "echo"
+    if os.environ.get("WSAI_BRAIN", "claude").lower() not in ("none", "echo"):
+        try:
+            from .backends.claude import ClaudeBrain
+            model = os.environ.get("WSAI_BRAIN_MODEL", "claude-sonnet-4-5")
+            brain = ClaudeBrain(model=model)
+            brain_name = "claude"
+        except Exception as exc:  # noqa: BLE001
+            logging.getLogger("wsai").warning("brain disabled (echo fallback): %s", exc)
+
+    monitor = Monitor()
+    stt = WhisperSTT()
+    tts = MeloTTS()
+    dash = Dashboard(monitor, host=host, port=port, stt=stt, tts=tts, brain=brain)
+    dash.start()
+    monitor.set_components({"source": "none", "vision": "none", "stt": "whisper",
+                            "brain": brain_name, "tts": "melo"})
+    monitor.set_status(running=True, listening=False)
+    monitor.log("info", "디스코드 음성 서버 시작 — STT+TTS GPU 워밍업 중…")
+    print("\n  STT+TTS 워밍업 중… (모델 로드 + CUDA 예열)")
+    dash.warm()
+    sdev = getattr(stt, "resolved_device", None) or "?"
+    monitor.set_status(listening=True)
+    monitor.log("info", f"음성 서버 준비 완료 (STT device={sdev}). 디스코드 봇 연결 대기.")
+
+    shown = host if host not in ("0.0.0.0", "") else _lan_ip()
+    print(f"\n  음성 서버 준비 완료 (STT device: {sdev}, 두뇌: {brain_name})")
+    print(f"  대시보드/상태:   http://{shown}:{port}")
+    print(f"  봇 연결 엔드포인트: http://127.0.0.1:{port}/api/voice-turn\n")
+    try:
+        while True:
+            time.sleep(3600)
+    except KeyboardInterrupt:
+        pass
+    finally:
+        dash.stop()
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(prog="wsai")
+    ap.add_argument("--voice", action="store_true", help="eyes-free voice loop (STT -> Brain -> TTS)")
+    ap.add_argument("--dashboard", action="store_true", help="serve the live status website")
+    ap.add_argument("--dashboard-loop-demo", action="store_true",
+                    help="keep generating mock demo utterances forever (off by default)")
+    ap.add_argument("--stt-test", action="store_true",
+                    help="serve the dashboard with a live GPU STT recognition test")
+    ap.add_argument("--voice-server", action="store_true",
+                    help="serve STT+TTS voice-turn endpoint for the Discord bot (dave/bot.mjs)")
+    ap.add_argument("--live", action="store_true", help="capture screen + Claude backends")
+    ap.add_argument("--env", action="store_true", help="build from WSAI_* env vars")
+    ap.add_argument("--port", type=int, default=int(os.environ.get("WSAI_DASHBOARD_PORT", "8787")),
+                    help="dashboard port (default 8787, or WSAI_DASHBOARD_PORT)")
+    ap.add_argument("--host", default=os.environ.get("WSAI_DASHBOARD_HOST", "0.0.0.0"),
+                    help="dashboard bind host (default 0.0.0.0)")
+    ap.add_argument("-v", "--verbose", action="store_true")
+    args = ap.parse_args()
+
+    logging.basicConfig(
+        level=logging.DEBUG if args.verbose else logging.INFO,
+        format="%(levelname)s %(name)s: %(message)s",
+    )
+
+    if args.stt_test:
+        _run_stt_test(args.host, args.port)
+        return
+
+    if args.voice_server:
+        _run_voice_server(args.host, args.port)
+        return
+
+    if args.dashboard:
+        # Default to the eyes-free voice preset for the demo; env can override.
+        settings = Settings.from_env() if args.env else Settings.voice()
+        demo = not args.env
+    elif args.voice:
+        settings, demo = Settings.voice(), True
+    elif args.live:
+        settings, demo = Settings.live(), False
+    elif args.env:
+        settings, demo = Settings.from_env(), False
+    else:
+        settings, demo = Settings.mock(), True
+
+    monitor: Monitor | None = None
+    dash = None
+    if args.dashboard:
+        from .dashboard import Dashboard
+
+        monitor = Monitor()
+        dash = Dashboard(monitor, host=args.host, port=args.port)
+        dash.start()
+        shown = args.host if args.host not in ("0.0.0.0", "") else _lan_ip()
+        print(f"\n  실시간 상태 사이트:  http://{shown}:{args.port}")
+        print(f"  (로컬:              http://127.0.0.1:{args.port} )\n")
+
+    try:
+        asyncio.run(
+            _run(
+                settings,
+                demo,
+                monitor,
+                demo_loop=args.dashboard and args.dashboard_loop_demo,
+                keep_dashboard_open=args.dashboard and demo and not args.dashboard_loop_demo,
+            )
+        )
+    except KeyboardInterrupt:
+        pass
+    finally:
+        if dash is not None:
+            dash.stop()
+
+
+if __name__ == "__main__":
+    main()

wsai/backends/capture_mss.py

@@ -0,0 +1,67 @@
+"""Local screen capture via `mss`.
+
+This is the practical "eye": run this on the machine that is in the Discord call
+viewing the shared screen, and it captures that monitor/region. Swap in a
+discord-web capture later without touching the pipeline.
+
+Requires: pip install mss pillow
+"""
+
+from __future__ import annotations
+
+import asyncio
+import io
+import time
+from typing import AsyncIterator
+
+from ..interfaces import Frame
+
+
+class MSSFrameSource:
+    def __init__(
+        self,
+        *,
+        monitor: int = 1,
+        region: dict | None = None,
+        interval: float = 1.5,
+        max_width: int = 1280,
+    ) -> None:
+        # `region` overrides `monitor`: {"top":.., "left":.., "width":.., "height":..}
+        self.monitor = monitor
+        self.region = region
+        self.interval = interval
+        self.max_width = max_width
+        self._sct = None
+
+    def _ensure(self):
+        if self._sct is None:
+            import mss  # lazy import so mock mode needs no dependency
+
+            self._sct = mss.mss()
+        return self._sct
+
+    def _grab_png(self) -> tuple[bytes, int, int]:
+        from PIL import Image
+
+        sct = self._ensure()
+        area = self.region or sct.monitors[self.monitor]
+        shot = sct.grab(area)
+        img = Image.frombytes("RGB", shot.size, shot.rgb)
+        if img.width > self.max_width:
+            h = int(img.height * self.max_width / img.width)
+            img = img.resize((self.max_width, h))
+        buf = io.BytesIO()
+        img.save(buf, format="PNG")
+        return buf.getvalue(), img.width, img.height
+
+    async def frames(self) -> AsyncIterator[Frame]:
+        while True:
+            # mss is blocking; keep the event loop free.
+            data, w, h = await asyncio.to_thread(self._grab_png)
+            yield Frame(data=data, width=w, height=h, ts=time.monotonic(), mime="image/png")
+            await asyncio.sleep(self.interval)
+
+    async def aclose(self) -> None:
+        if self._sct is not None:
+            self._sct.close()
+            self._sct = None

wsai/backends/claude.py

@@ -0,0 +1,158 @@
+"""Cloud brain + vision via the Anthropic (Claude) API.
+
+Both share one auth resolver. Vision sends the frame as a base64 image; the
+brain is a plain chat call that receives the latest screen description as
+context.
+
+Auth (two ways, tried in this order):
+  1. ANTHROPIC_API_KEY  -> standard API-key auth.
+  2. A Claude Code OAuth token (this deployment's Max login) read from the
+     credentials file at $CLAUDE_CREDENTIALS_PATH. OAuth tokens require the
+     first system block to be exactly the Claude Code identity string and are
+     sent as a Bearer token (auth_token=), not x-api-key. The token is
+     re-read before each request so a refresh rotated into the file by the
+     host is picked up without a restart.
+
+Requires: pip install anthropic
+"""
+
+from __future__ import annotations
+
+import base64
+import json
+import os
+import time
+
+from ..interfaces import Frame, Reply, ScreenObservation
+
+# Claude Code OAuth tokens only answer when the first system block is exactly
+# this identity string; the real persona/instructions go in later blocks.
+_CLAUDE_CODE_ID = "You are Claude Code, Anthropic's official CLI for Claude."
+
+# Claude occasionally returns 529 Overloaded; the anthropic SDK retries >=500
+# (and 429) with exponential backoff, but its default of 2 tries can be too few
+# to ride out a busy window. Bump it so a transient overload doesn't drop the
+# voice turn to the apology fallback. Kept modest so a *sustained* overload
+# still fails fast rather than leaving the bot silent for many seconds.
+_MAX_RETRIES = int(os.environ.get("WSAI_BRAIN_MAX_RETRIES", "4"))
+
+
+def _load_oauth_token() -> str | None:
+    path = os.environ.get("CLAUDE_CREDENTIALS_PATH")
+    if not path or not os.path.exists(path):
+        return None
+    try:
+        with open(path) as f:
+            return json.load(f)["claudeAiOauth"]["accessToken"]
+    except (OSError, KeyError, ValueError):
+        return None
+
+
+class _Auth:
+    """Resolves a Claude client, preferring an explicit/env API key and falling
+    back to the deployment's OAuth token. The OAuth client is rebuilt whenever
+    the token in the credentials file changes (host-side refresh)."""
+
+    def __init__(self, api_key: str | None = None) -> None:
+        import anthropic  # lazy so mock mode needs no dependency
+
+        self._anthropic = anthropic
+        self._explicit_key = api_key
+        self._client = None
+        self._token: str | None = None
+        self._oauth = False
+
+    def client(self):
+        key = self._explicit_key or os.environ.get("ANTHROPIC_API_KEY")
+        if key:
+            if self._client is None:
+                self._client = self._anthropic.AsyncAnthropic(api_key=key, max_retries=_MAX_RETRIES)
+            self._oauth = False
+            return self._client
+        tok = _load_oauth_token()
+        if not tok:
+            raise RuntimeError(
+                "No Claude auth: set ANTHROPIC_API_KEY or provide a Claude "
+                "OAuth login via CLAUDE_CREDENTIALS_PATH."
+            )
+        if tok != self._token:
+            self._token = tok
+            self._client = self._anthropic.AsyncAnthropic(auth_token=tok, max_retries=_MAX_RETRIES)
+        self._oauth = True
+        return self._client
+
+    def system(self, *blocks: str) -> list[dict]:
+        """Build the system prompt, prepending the Claude Code identity block
+        when authing via OAuth (required) — harmless to include either way."""
+        texts = [_CLAUDE_CODE_ID, *blocks] if self._oauth else list(blocks)
+        return [{"type": "text", "text": t} for t in texts if t]
+
+
+class ClaudeVision:
+    def __init__(self, *, model: str = "claude-sonnet-4-5", api_key: str | None = None) -> None:
+        self.model = model
+        self._auth = _Auth(api_key)
+
+    async def describe(self, frame: Frame, hint: str | None = None) -> ScreenObservation:
+        prompt = hint or (
+            "이건 디스코드 화면공유 캡처야. 지금 화면에서 무슨 일이 벌어지는지 "
+            "2~3문장으로 한국어로 간결하게 설명해줘. 코드/에러/게임/문서 등 맥락을 짚어줘."
+        )
+        b64 = base64.b64encode(frame.data).decode()
+        client = self._auth.client()
+        resp = await client.messages.create(
+            model=self.model,
+            max_tokens=300,
+            system=self._auth.system(),
+            messages=[
+                {
+                    "role": "user",
+                    "content": [
+                        {
+                            "type": "image",
+                            "source": {"type": "base64", "media_type": frame.mime, "data": b64},
+                        },
+                        {"type": "text", "text": prompt},
+                    ],
+                }
+            ],
+        )
+        text = "".join(b.text for b in resp.content if b.type == "text")
+        return ScreenObservation(text=text.strip(), ts=frame.ts)
+
+
+class ClaudeBrain:
+    PERSONA = (
+        "너는 사용자의 디스코드 화면공유를 실시간으로 함께 보는 AI 파트너야. "
+        "화면 설명을 참고해 자연스러운 반말/존댓말은 사용자에 맞추고, 짧고 대화하듯 답해. "
+        "화면을 못 봤으면 솔직히 말해. "
+        "네 답변은 음성으로 읽히니 마크다운·코드블록·백틱 같은 서식 없이 평범한 말로만 답해. "
+        "감정은 대괄호 태그로 표현해. 태그 자체는 소리로 읽히지 않고, 그 뒤 문장의 목소리 톤(피치·속도)을 바꿔줘. "
+        "답변 맨 앞에 감정 태그 하나로 시작하고, 답변 도중 감정이 바뀌면 그 지점에 새 태그를 넣어. "
+        "예: [속상함] 정말 힘들었겠다. [힘차게] 하지만 넌 할 수 있어! "
+        "쓸 수 있는 감정: 기쁨, 신남, 희망(힘차게), 슬픔(속상함), 화남, 두려움, 놀람, 차분, 다정, 진지, 실망, 피곤, "
+        "사랑스럽게, 장난스럽게(웃으며), 속삭임, 외침, 단호, 안도, 궁금, 반가움. "
+        "감정 태그가 아닌 진짜 대괄호 내용(예: [1번], [메모])은 그대로 읽히니 필요하면 그렇게 써도 돼. "
+        "답변은 최대한 짧고 간결하게, 한두 문장 이내로 해."
+    )
+
+    def __init__(self, *, model: str = "claude-sonnet-4-5", api_key: str | None = None) -> None:
+        self.model = model
+        self._auth = _Auth(api_key)
+
+    async def respond(self, user_text, screen, history) -> Reply:
+        msgs = []
+        for user, ai in history:
+            msgs.append({"role": "user", "content": user})
+            msgs.append({"role": "assistant", "content": ai})
+        screen_note = f"[지금 화면] {screen.text}\n\n" if screen else "[지금 화면] (아직 못 읽음)\n\n"
+        msgs.append({"role": "user", "content": screen_note + user_text})
+        client = self._auth.client()
+        resp = await client.messages.create(
+            model=self.model,
+            max_tokens=400,
+            system=self._auth.system(self.PERSONA),
+            messages=msgs,
+        )
+        text = "".join(b.text for b in resp.content if b.type == "text")
+        return Reply(text=text.strip(), ts=time.monotonic())

wsai/backends/emotion.py

@@ -0,0 +1,139 @@
+"""Emotion tags for expressive TTS.
+
+Claude can sprinkle ``[감정]`` tags through a reply to colour the delivery, e.g.
+
+    "[속상함] 정말 힘들었겠다. [힘차게] 하지만 넌 할 수 있어!"
+
+An emotion tag is NOT spoken — instead it shifts the *following* text's pitch and
+speed until the next tag. A bracketed word that is NOT a known emotion is left as
+ordinary spoken content (the brackets are dropped, the words are read).
+
+The emotion vocabulary is grounded in the de-facto industry set used by Azure
+Neural TTS speaking styles (cheerful, sad, angry, excited, friendly, hopeful,
+terrified, shouting, whispering) together with Ekman's six basic emotions
+(happiness, sadness, anger, fear, surprise, disgust). Each canonical emotion maps
+to a ``(speed_multiplier, pitch_semitones)`` pair; the multiplier scales the base
+synthesis speed and the semitone offset is applied as a pitch shift on the wav.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+
+# Canonical emotion -> (speed multiplier relative to base, pitch shift in semitones).
+# Kept deliberately modest so delivery stays natural, not cartoonish.
+EMOTION_PARAMS: dict[str, tuple[float, float]] = {
+    "happy":       (1.08,  2.0),   # 기쁨 / cheerful
+    "excited":     (1.15,  3.0),   # 신남 / excited
+    "hopeful":     (1.10,  1.5),   # 희망 / 힘차게
+    "sad":         (0.90, -2.5),   # 슬픔 / sad
+    "angry":       (1.12,  1.0),   # 화남 / angry
+    "fearful":     (1.12,  2.0),   # 두려움 / terrified
+    "surprised":   (1.05,  3.0),   # 놀람 / surprise
+    "disgust":     (0.96, -1.0),   # 혐오 / disgust
+    "calm":        (0.95, -1.0),   # 차분 / calm
+    "friendly":    (1.00,  1.0),   # 다정 / friendly
+    "serious":     (0.97, -1.0),   # 진지 / serious
+    "disappointed":(0.92, -2.0),   # 실망 / disappointed
+    "tired":       (0.90, -2.0),   # 피곤 / 지침
+    "affectionate":(0.98,  1.0),   # 사랑스럽게 / affectionate
+    "playful":     (1.08,  2.0),   # 장난스럽게 / playful
+    "whisper":     (0.92, -1.5),   # 속삭임 / whispering
+    "shout":       (1.05,  2.5),   # 외침 / shouting
+    "determined":  (1.05,  0.5),   # 단호 / determined
+    "relieved":    (0.95,  0.5),   # 안도 / relieved
+    "curious":     (1.03,  1.5),   # 궁금 / curious
+}
+
+# Every spelling Claude might realistically emit, mapped to a canonical emotion.
+# False negatives (reading an emotion word aloud) are harmless; false positives
+# (silently dropping real content) are not — so match spellings exactly rather
+# than fuzzily.
+_SYNONYMS: dict[str, str] = {}
+
+
+def _register(canonical: str, *words: str) -> None:
+    for w in words:
+        _SYNONYMS[_norm(w)] = canonical
+
+
+def _norm(word: str) -> str:
+    # Compare on a squeezed, lower-cased form so "힘 차게" == "힘차게".
+    return re.sub(r"\s+", "", word).lower()
+
+
+_register("happy", "기쁨", "기쁘게", "기뻐", "기뻐하며", "행복", "행복하게", "행복하게도", "즐겁게", "즐거움", "밝게", "반가움", "반갑게", "반가워", "cheerful", "happy", "joyful")
+_register("excited", "신남", "신나게", "신나서", "흥분", "들뜬", "들떠서", "설렘", "설레며", "excited", "thrilled")
+_register("hopeful", "희망", "희망차게", "힘차게", "힘내", "힘내서", "응원", "응원하며", "격려", "hopeful", "encouraging")
+_register("sad", "슬픔", "슬프게", "슬퍼", "슬퍼하며", "속상함", "속상하게", "속상해", "우울", "우울하게", "안타깝게", "울먹이며", "sad", "sorrowful")
+_register("angry", "화남", "화나게", "화나서", "화가남", "분노", "분노하며", "짜증", "짜증내며", "angry", "furious")
+_register("fearful", "두려움", "두렵게", "무섭게", "무서워하며", "불안", "불안하게", "겁먹은", "겁먹고", "떨리는", "fearful", "terrified", "anxious")
+_register("surprised", "놀람", "놀라며", "놀랍게", "놀라서", "깜짝", "경악", "surprised", "shocked")
+_register("disgust", "혐오", "역겹게", "질색", "disgust", "disgusted")
+_register("calm", "차분", "차분하게", "침착", "침착하게", "담담하게", "잔잔하게", "calm", "gentle")
+_register("friendly", "다정", "다정하게", "친근", "친근하게", "부드럽게", "따뜻하게", "friendly", "warm")
+_register("serious", "진지", "진지하게", "무겁게", "엄숙하게", "serious", "solemn")
+_register("disappointed", "실망", "실망스럽게", "실망하며", "낙담", "disappointed")
+_register("tired", "피곤", "피곤하게", "지침", "지쳐서", "지친", "힘없이", "tired", "weary", "exhausted")
+_register("affectionate", "사랑스럽게", "애정", "애정어린", "다정스럽게", "affectionate", "loving")
+_register("playful", "장난스럽게", "장난치며", "유쾌하게", "익살스럽게", "웃으며", "웃으면서", "playful", "teasing")
+_register("curious", "궁금", "궁금하게", "궁금해하며", "궁금해서", "호기심", "curious", "inquisitive")
+_register("whisper", "속삭임", "속삭이며", "조용히", "나지막이", "whisper", "whispering")
+_register("shout", "외침", "외치며", "큰소리로", "소리치며", "우렁차게", "shout", "shouting")
+_register("determined", "단호", "단호하게", "결연하게", "당당하게", "determined", "confident")
+_register("relieved", "안도", "안도하며", "안심", "안심하며", "relieved")
+
+
+def match_emotion(inner: str) -> str | None:
+    """Return the canonical emotion for a bracket's inner text, or None if the
+    text is not a recognised emotion word (and should therefore be spoken)."""
+    return _SYNONYMS.get(_norm(inner))
+
+
+@dataclass
+class Segment:
+    text: str
+    speed: float
+    pitch: float  # semitones; 0.0 == no shift
+
+
+_TAG_RE = re.compile(r"\[([^\[\]]*)\]")
+
+
+def parse_segments(text: str, base_speed: float) -> list[Segment]:
+    """Split ``text`` into consecutive spoken segments, each carrying the speed
+    and pitch implied by the most recent emotion tag.
+
+    * An emotion tag switches the active emotion for everything after it and is
+      not spoken.
+    * A non-emotion bracket keeps its inner words as spoken text (brackets gone).
+    * Text before any tag is spoken with neutral delivery (base speed, no shift).
+    """
+    segments: list[Segment] = []
+    cur_speed, cur_pitch = base_speed, 0.0
+    buf: list[str] = []
+
+    def flush() -> None:
+        joined = "".join(buf).strip()
+        if joined:
+            segments.append(Segment(joined, cur_speed, cur_pitch))
+        buf.clear()
+
+    pos = 0
+    for m in _TAG_RE.finditer(text):
+        emotion = match_emotion(m.group(1))
+        buf.append(text[pos:m.start()])
+        pos = m.end()
+        if emotion is None:
+            # Not an emotion — read the bracket's contents, drop the brackets.
+            buf.append(m.group(1))
+        else:
+            # Emotion tag — everything so far belongs to the previous emotion;
+            # flush it, then switch delivery for what follows.
+            flush()
+            mult, semis = EMOTION_PARAMS[emotion]
+            cur_speed, cur_pitch = base_speed * mult, semis
+    buf.append(text[pos:])
+    flush()
+    return segments  # empty when there is nothing speakable (blank or all-tags)

wsai/backends/melo.py

@@ -0,0 +1,225 @@
+"""Real Korean TTS via MeloTTS, run as a persistent out-of-venv worker.
+
+MeloTTS needs its own interpreter (melo311). Loading the model costs several
+seconds, so we keep one worker process alive and stream synthesis requests to
+it (see melo_worker.py for the protocol). Each `speak()` writes a wav to
+`out_dir` and hands the path to a sink (default: log it). The Discord voice
+integration later swaps the sink for "play this wav into the call".
+
+Env:
+    WSAI_MELO_PYTHON   interpreter with melo installed
+                       (default: /home/claude/jarvis-tts/melo311/bin/python)
+    WSAI_MELO_DEVICE   cpu | cuda | auto  (default auto: GPU if torch sees one,
+                       else CPU; the worker falls back to CPU if CUDA fails)
+    WSAI_TTS_OUT_DIR   where wavs are written (default ~/.cache/wsai/tts)
+    WSAI_TTS_SPEED     synthesis speed multiplier (default 1.3)
+"""
+
+from __future__ import annotations
+
+import asyncio
+import collections
+import json
+import logging
+import os
+import re
+import time
+from pathlib import Path
+from typing import Awaitable, Callable
+
+from ..interfaces import Reply
+from .emotion import parse_segments
+
+log = logging.getLogger("wsai.tts.melo")
+
+_DEFAULT_PYTHON = "/home/claude/jarvis-tts/melo311/bin/python"
+
+_FENCE_RE = re.compile(r"```[^\n`]*\n?(.*?)```", re.DOTALL)
+_LINK_RE = re.compile(r"\[([^\]]+)\]\([^)]*\)")
+_INLINE_CODE_RE = re.compile(r"`+([^`]*)`+")
+
+
+def normalize_for_speech(text: str) -> str:
+    """Flatten Claude's markdown/code formatting into plain prose before TTS.
+
+    MeloTTS's Korean text normaliser has no dictionary entry for characters
+    like the backtick and dies with ``KeyError: '`'`` — which crashes the whole
+    voice turn the moment the model mentions a command or shows a code block.
+    Code/markdown also reads terribly aloud. So strip the formatting and keep
+    the words. Every spoken path (dashboard voice turn and the Discord speak()
+    bridge) funnels through ``synth()``, so normalising there covers them both.
+    """
+    if not text:
+        return text
+    # Fenced code block -> keep its inner text as spoken words, drop the fences.
+    text = _FENCE_RE.sub(lambda m: " " + m.group(1) + " ", text)
+    # [label](url) -> label
+    text = _LINK_RE.sub(r"\1", text)
+    # `code` -> code
+    text = _INLINE_CODE_RE.sub(r"\1", text)
+    # Any stray/unbalanced backtick that survived -> gone. This is the exact
+    # character that crashes MeloTTS, so guarantee none remain.
+    text = text.replace("`", "")
+    # Markdown structure markers -> plain text.
+    text = re.sub(r"(?m)^\s{0,3}#{1,6}\s*", "", text)   # ATX headings
+    text = re.sub(r"(?m)^\s{0,3}>\s?", "", text)         # blockquotes
+    text = re.sub(r"(?m)^\s{0,3}[-*+]\s+", "", text)     # bullet list markers
+    text = re.sub(r"[*_]{1,3}", "", text)                # bold/italic emphasis
+    # Collapse the whitespace the stripping leaves behind.
+    text = re.sub(r"[ \t]+", " ", text)
+    text = re.sub(r"\n{2,}", "\n", text)
+    return text.strip()
+
+# A sink receives the finished wav path plus the reply it voices.
+Sink = Callable[[str, Reply], Awaitable[None]]
+
+
+async def _log_sink(path: str, reply: Reply) -> None:
+    log.info("TTS wav ready: %s (%s)", path, reply.text[:40])
+
+
+class MeloTTS:
+    def __init__(
+        self,
+        *,
+        python: str | None = None,
+        device: str | None = None,
+        out_dir: str | None = None,
+        speed: float | None = None,
+        sink: Sink | None = None,
+    ) -> None:
+        self.python = python or os.environ.get("WSAI_MELO_PYTHON", _DEFAULT_PYTHON)
+        self.device = device or os.environ.get("WSAI_MELO_DEVICE", "auto")
+        self.out_dir = Path(out_dir or os.environ.get("WSAI_TTS_OUT_DIR")
+                            or (Path.home() / ".cache/wsai/tts"))
+        self.speed = float(speed if speed is not None
+                           else os.environ.get("WSAI_TTS_SPEED", "1.3"))
+        self.sink = sink or _log_sink
+        self._proc: asyncio.subprocess.Process | None = None
+        self._lock = asyncio.Lock()
+        self._n = 0
+        self.load_ms: int | None = None
+        # Keep the worker's most recent stderr lines so a crash reports its real
+        # cause instead of a bare JSONDecodeError. Bounded so it can't grow.
+        self._stderr_tail: collections.deque[str] = collections.deque(maxlen=40)
+        self._stderr_task: asyncio.Task | None = None
+
+    async def _drain_stderr(self, stream: asyncio.StreamReader) -> None:
+        # The worker redirects fd1 -> fd2, so ALL library chatter lands on
+        # stderr. If we PIPE stderr but never read it, the OS pipe buffer fills
+        # and the worker blocks forever. So we continuously drain it and keep
+        # only the last few lines for diagnostics.
+        try:
+            while True:
+                line = await stream.readline()
+                if not line:
+                    return
+                self._stderr_tail.append(line.decode(errors="replace").rstrip())
+        except asyncio.CancelledError:
+            raise
+        except Exception:  # draining must never crash the caller
+            return
+
+    def _stderr_hint(self) -> str:
+        tail = "\n".join(self._stderr_tail)
+        return f" worker stderr tail:\n{tail}" if tail else " (worker produced no stderr)"
+
+    async def warmup(self) -> None:
+        """Start and load the worker now so the first real utterance is warm.
+
+        Called at pipeline startup so users don't wait ~7s (CPU model load) for
+        the very first spoken reply."""
+        await self._ensure()
+
+    async def _ensure(self) -> None:
+        if self._proc is not None and self._proc.returncode is None:
+            return
+        self.out_dir.mkdir(parents=True, exist_ok=True)
+        env = {**os.environ, "WSAI_MELO_DEVICE": self.device}
+        # Run the worker module from the wsai source tree with the melo venv.
+        repo_root = str(Path(__file__).resolve().parents[2])
+        self._proc = await asyncio.create_subprocess_exec(
+            self.python, "-m", "wsai.backends.melo_worker",
+            cwd=repo_root, env=env,
+            stdin=asyncio.subprocess.PIPE,
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+        )
+        self._stderr_tail.clear()
+        assert self._proc.stderr is not None
+        self._stderr_task = asyncio.create_task(self._drain_stderr(self._proc.stderr))
+        ready = await self._proc.stdout.readline()
+        if not ready:  # worker died before signalling ready
+            await self._proc.wait()
+            raise RuntimeError(
+                f"melo worker exited before ready (code {self._proc.returncode})."
+                f"{self._stderr_hint()}"
+            )
+        try:
+            info = json.loads(ready.decode())
+        except json.JSONDecodeError as exc:
+            raise RuntimeError(
+                f"melo worker sent invalid ready line {ready!r}: {exc}."
+                f"{self._stderr_hint()}"
+            ) from exc
+        if not info.get("ready"):
+            raise RuntimeError(
+                f"melo worker failed to start: {info}.{self._stderr_hint()}"
+            )
+        self.load_ms = info.get("ms")
+        log.info("melo worker ready in %s ms on %s", self.load_ms, info.get("device"))
+
+    async def synth(self, text: str) -> str:
+        """Synthesize `text` to a wav and return its path (no sink). Reusable by
+        callers that want the wav directly (e.g. the Discord voice bridge)."""
+        await self._ensure()
+        text = normalize_for_speech(text)
+        # Split on [감정] tags: each tag steers pitch/speed for the text that
+        # follows (and is itself not spoken); non-emotion brackets stay as words.
+        segments = parse_segments(text, self.speed)
+        self._n += 1
+        out = str(self.out_dir / f"tts-{self._n:06d}.wav")
+        if segments:
+            payload = {
+                "segments": [
+                    {"text": s.text, "speed": s.speed, "pitch": s.pitch}
+                    for s in segments
+                ],
+                "out": out,
+            }
+        else:  # empty/whitespace reply: keep legacy single-utterance behaviour
+            payload = {"text": text, "out": out, "speed": self.speed}
+        req = json.dumps(payload)
+        s = time.monotonic()
+        async with self._lock:
+            assert self._proc and self._proc.stdin and self._proc.stdout
+            self._proc.stdin.write((req + "\n").encode())
+            await self._proc.stdin.drain()
+            resp = await self._proc.stdout.readline()
+        if not resp:
+            raise RuntimeError(f"melo worker closed unexpectedly.{self._stderr_hint()}")
+        res = json.loads(resp.decode())
+        if not res.get("ok"):
+            raise RuntimeError(f"melo synth failed: {res.get('error')}")
+        log.debug("synth %d ms (worker %s ms)", int((time.monotonic() - s) * 1000), res.get("ms"))
+        return res["out"]
+
+    async def speak(self, reply: Reply) -> None:
+        out = await self.synth(reply.text)
+        await self.sink(out, reply)
+
+    async def aclose(self) -> None:
+        if self._proc is not None and self._proc.returncode is None:
+            try:
+                self._proc.terminate()
+                await asyncio.wait_for(self._proc.wait(), timeout=5)
+            except (ProcessLookupError, asyncio.TimeoutError):
+                pass
+        if self._stderr_task is not None:
+            self._stderr_task.cancel()
+            try:
+                await self._stderr_task
+            except (asyncio.CancelledError, Exception):
+                pass
+            self._stderr_task = None
+        self._proc = None

wsai/backends/melo_worker.py

@@ -0,0 +1,156 @@
+"""Persistent MeloTTS worker (Korean).
+
+MeloTTS lives in its own Python (melo311); loading the model takes seconds, so
+we load it ONCE here and then serve synthesis requests over stdin/stdout. This
+process is launched with the melo311 interpreter by wsai.backends.melo.MeloTTS.
+
+MeloTTS (and its deps) print progress straight to stdout, which would corrupt
+the JSON protocol. So on startup we split the streams: a private duplicate of
+the original stdout carries the protocol, and fd 1 is redirected to fd 2 so all
+library chatter lands on stderr instead.
+
+Protocol (one JSON object per line, on the protocol channel):
+    <- {"text": "...", "out": "/abs/path.wav", "speed": 1.3}
+    <- {"segments": [{"text": "...", "speed": 1.3, "pitch": 2.0}, ...],
+        "out": "/abs/path.wav"}   # expressive form: per-segment speed + pitch
+    -> {"ok": true, "out": "/abs/path.wav", "ms": 123}
+    -> {"ok": false, "error": "..."}
+On startup, once the model is ready, it emits exactly one line:
+    -> {"ready": true, "ms": <load-ms>, "device": "cpu"}
+
+``pitch`` is a semitone offset applied to that segment's wav (0 == no shift) so
+emotion tags can raise/lower the voice without changing the words. Segments are
+synthesised independently and concatenated with a short gap so a single reply can
+carry several emotions.
+"""
+
+import json
+import os
+import sys
+import time
+
+# Split protocol from library noise BEFORE importing anything heavy.
+_proto = os.fdopen(os.dup(1), "w", buffering=1)  # private copy of real stdout
+os.dup2(2, 1)  # fd1 -> stderr, so stray library prints don't hit the protocol
+
+
+def _emit(obj: dict) -> None:
+    _proto.write(json.dumps(obj) + "\n")
+    _proto.flush()
+
+
+def _log(*a):
+    print(*a, file=sys.stderr, flush=True)
+
+
+def main() -> None:
+    lang = "KR"
+    requested = os.environ.get("WSAI_MELO_DEVICE", "auto")  # cpu | cuda | auto
+    from melo.api import TTS  # heavy import; only in the melo venv
+
+    def _has_cuda() -> bool:
+        try:
+            import torch
+
+            return torch.cuda.is_available()
+        except Exception:
+            return False
+
+    device = requested
+    if requested == "auto":
+        device = "cuda" if _has_cuda() else "cpu"
+
+    t0 = time.monotonic()
+    try:
+        tts = TTS(language=lang, device=device)
+    except Exception as exc:
+        # CUDA picked but unusable (CPU-only torch, missing libs, OOM): fall back
+        # to CPU rather than leaving the whole voice loop dead.
+        if device == "cuda":
+            _log(f"[melo_worker] CUDA load failed ({exc}); falling back to CPU")
+            device = "cpu"
+            tts = TTS(language=lang, device=device)
+        else:
+            raise
+    speaker_id = tts.hps.data.spk2id[lang]
+    sr = tts.hps.data.sampling_rate
+    load_ms = int((time.monotonic() - t0) * 1000)
+
+    import numpy as np
+    import soundfile
+
+    _GAP = np.zeros(int(sr * 0.12), dtype=np.float32)  # 120 ms between segments
+
+    def _pitch_shift(audio, semitones: float):
+        if not semitones:
+            return audio
+        import librosa
+
+        return librosa.effects.pitch_shift(
+            audio.astype(np.float32), sr=sr, n_steps=float(semitones)
+        )
+
+    def _synth_segments(segments: list[dict], out: str) -> None:
+        """Synthesize each segment, pitch-shift it, and concatenate to one wav."""
+        pieces = []
+        for i, seg in enumerate(segments):
+            text = seg["text"]
+            if not text.strip():
+                continue
+            speed = float(seg.get("speed", 1.0))
+            pitch = float(seg.get("pitch", 0.0))
+            audio = tts.tts_to_file(text, speaker_id, None, speed=speed)
+            audio = _pitch_shift(np.asarray(audio, dtype=np.float32), pitch)
+            if pieces:
+                pieces.append(_GAP)
+            pieces.append(audio)
+        if not pieces:
+            raise ValueError("no speakable segment")
+        soundfile.write(out, np.concatenate(pieces), sr)
+
+    # Warm up before signalling ready: the first CUDA synth pays a large lazy
+    # cost (kernel autotune/cudnn), ~10s cold vs ~130ms hot, which would blow the
+    # voice loop's ~1s budget on the very first reply. Do that dummy synth here so
+    # "ready" means "hot". Failures must not block startup.
+    warmup_ms = None
+    try:
+        warm_out = os.path.expanduser("~/.cache/wsai/tts/_warmup.wav")
+        os.makedirs(os.path.dirname(warm_out), exist_ok=True)
+        w = time.monotonic()
+        tts.tts_to_file("워밍업", speaker_id, warm_out, speed=1.3)
+        # Also JIT-warm librosa's pitch shifter (first call pays ~0.4s numba
+        # compile) so the first *emotional* reply doesn't stall.
+        import librosa
+
+        librosa.effects.pitch_shift(np.zeros(sr, dtype=np.float32), sr=sr, n_steps=1.0)
+        warmup_ms = int((time.monotonic() - w) * 1000)
+    except Exception as exc:
+        _log(f"[melo_worker] warmup skipped: {exc}")
+
+    _emit({"ready": True, "ms": load_ms, "device": device, "warmup_ms": warmup_ms})
+    _log(f"[melo_worker] model ready in {load_ms} ms on {device} (warmup {warmup_ms} ms)")
+
+    for line in sys.stdin:
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            req = json.loads(line)
+            out = req["out"]
+            if out.startswith("/tmp") or out.startswith("/dev/shm"):
+                raise ValueError(f"refusing RAM-backed tmpfs path: {out}")
+            s = time.monotonic()
+            if "segments" in req:
+                _synth_segments(req["segments"], out)
+            else:  # legacy single-utterance form
+                speed = float(req.get("speed", 1.0))
+                tts.tts_to_file(req["text"], speaker_id, out, speed=speed)
+            ms = int((time.monotonic() - s) * 1000)
+            _emit({"ok": True, "out": out, "ms": ms})
+        except Exception as exc:  # keep the worker alive across bad requests
+            _emit({"ok": False, "error": f"{type(exc).__name__}: {exc}"})
+            _log(f"[melo_worker] error: {exc}")
+
+
+if __name__ == "__main__":
+    main()

wsai/backends/mock.py

@@ -0,0 +1,108 @@
+"""Mock backends. These let the full pipeline run with no GPU, no mic, no API
+key — so the skeleton is verifiable and gives every real backend a reference
+implementation to match.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import itertools
+import time
+from typing import AsyncIterator
+
+from ..interfaces import (
+    Frame,
+    Reply,
+    ScreenObservation,
+    Utterance,
+)
+
+
+class MockFrameSource:
+    """Emits tiny synthetic frames on a fixed interval."""
+
+    def __init__(self, interval: float = 1.0, limit: int | None = None) -> None:
+        self.interval = interval
+        self.limit = limit
+
+    async def frames(self) -> AsyncIterator[Frame]:
+        for i in itertools.count():
+            if self.limit is not None and i >= self.limit:
+                return
+            yield Frame(
+                data=b"\x89PNG\r\n\x1a\n",  # PNG magic; enough for a stub
+                width=1280,
+                height=720,
+                ts=time.monotonic(),
+                mime="image/png",
+            )
+            await asyncio.sleep(self.interval)
+
+    async def aclose(self) -> None:  # nothing to release
+        return
+
+
+class MockVision:
+    """Pretends to read the screen. Cycles through a few canned scenes."""
+
+    SCENES = [
+        "VS Code is open with a Python file; a traceback is visible in the terminal.",
+        "A browser shows a GitHub pull request diff.",
+        "A game is running; the player is in a menu screen.",
+    ]
+
+    def __init__(self) -> None:
+        self._i = 0
+
+    async def describe(self, frame: Frame, hint: str | None = None) -> ScreenObservation:
+        scene = self.SCENES[self._i % len(self.SCENES)]
+        self._i += 1
+        return ScreenObservation(text=scene, ts=frame.ts)
+
+
+class MockSTT:
+    """Feeds a scripted set of user utterances, then goes quiet."""
+
+    def __init__(
+        self,
+        script: list[str] | None = None,
+        interval: float = 2.0,
+        loop: bool = False,
+    ) -> None:
+        self.script = script or [
+            "지금 화면에 뭐 보여?",
+            "저 에러 왜 나는 거야?",
+            "고마워",
+        ]
+        self.interval = interval
+        self.loop = loop
+
+    async def utterances(self) -> AsyncIterator[Utterance]:
+        while True:
+            for line in self.script:
+                await asyncio.sleep(self.interval)
+                yield Utterance(text=line, ts=time.monotonic(), source="voice")
+            if not self.loop:
+                return
+
+    async def aclose(self) -> None:
+        return
+
+
+class MockTTS:
+    """'Speaks' by printing. Real TTS swaps in here."""
+
+    async def speak(self, reply: Reply) -> None:
+        print(f"[TTS] {reply.text}")
+
+
+class MockBrain:
+    """Echo-style brain that references the current screen, so you can see the
+    screen context actually reaching the conversation loop."""
+
+    async def respond(self, user_text, screen, history) -> Reply:
+        seen = screen.text if screen else "아직 화면을 못 읽었어요"
+        return Reply(
+            text=f'(화면: "{seen}") 라고 봤어요. 말씀하신 "{user_text}"에 대해 답하자면… [mock]',
+            ts=time.monotonic(),
+        )

wsai/backends/whisper.py

@@ -0,0 +1,213 @@
+"""Real STT via faster-whisper, run as a persistent out-of-venv worker.
+
+faster-whisper needs its own interpreter (whisper312) and loading the model
+costs several seconds, so we keep one worker process alive and stream
+transcription requests to it (see whisper_worker.py for the protocol) — the
+same warm-worker shape as MeloTTS on the TTS side.
+
+`transcribe(wav)` is the core engine: hand it a wav path, get the recognised
+text back. It is what closes the voice round trip (TTS wav -> STT text) and what
+the Discord voice path will call once per detected utterance.
+
+`utterances()` turns this into a SpeechToText source: it pulls finished-utterance
+wav paths from an injected `audio_source` and yields a transcribed Utterance for
+each. Until the Discord voice receiver is wired, `audio_source` is None and the
+loop simply idles (like the eyes-free perception loop), while `transcribe()`
+stays usable directly.
+
+Env:
+    WSAI_WHISPER_PYTHON    interpreter with faster-whisper installed
+                           (default: /home/claude/jarvis-stt/whisper312/bin/python)
+    WSAI_WHISPER_MODEL     model size/name (default: small)
+    WSAI_WHISPER_DEVICE    cpu | cuda | auto  (default auto: GPU if present,
+                           else CPU; the worker falls back to CPU if CUDA fails)
+    WSAI_WHISPER_LANGUAGE  forced language, e.g. ko (default ko; "" = autodetect)
+"""
+
+from __future__ import annotations
+
+import asyncio
+import collections
+import json
+import logging
+import os
+import time
+from pathlib import Path
+from typing import AsyncIterator
+
+from ..interfaces import Utterance
+
+log = logging.getLogger("wsai.stt.whisper")
+
+_DEFAULT_PYTHON = "/home/claude/jarvis-stt/whisper312/bin/python"
+
+
+def _cuda_lib_dirs(python_exe: str) -> list[str]:
+    """nvidia/*/lib dirs of the worker venv (cublas, cudnn, ...), for
+    LD_LIBRARY_PATH so ctranslate2 can dlopen the CUDA runtime. Empty if the
+    interpreter has no such packages (CPU-only install)."""
+    import glob
+
+    # Use the literal path, NOT .resolve(): the venv's bin/python is a symlink
+    # into the uv-managed interpreter, and resolving it would jump out of the
+    # venv and miss its site-packages/nvidia libs.
+    venv = Path(python_exe).parent.parent  # .../bin/python -> venv root
+    dirs = glob.glob(str(venv / "lib" / "python*" / "site-packages" / "nvidia" / "*" / "lib"))
+    return sorted(set(dirs))
+
+
+class WhisperSTT:
+    def __init__(
+        self,
+        *,
+        python: str | None = None,
+        model: str | None = None,
+        device: str | None = None,
+        language: str | None = None,
+        audio_source: AsyncIterator[str] | None = None,
+    ) -> None:
+        self.python = python or os.environ.get("WSAI_WHISPER_PYTHON", _DEFAULT_PYTHON)
+        self.model = model or os.environ.get("WSAI_WHISPER_MODEL", "small")
+        self.device = device or os.environ.get("WSAI_WHISPER_DEVICE", "auto")
+        # "" means autodetect; a real code like "ko" forces the language.
+        env_lang = os.environ.get("WSAI_WHISPER_LANGUAGE", "ko")
+        self.language = language if language is not None else (env_lang or None)
+        self.audio_source = audio_source
+        self._proc: asyncio.subprocess.Process | None = None
+        self._lock = asyncio.Lock()
+        self.load_ms: int | None = None
+        self.resolved_device: str | None = None  # "cuda" | "cpu", known after start
+        # Keep the worker's most recent stderr so a crash reports its real cause
+        # instead of a bare JSONDecodeError. Bounded so it can't grow unbounded.
+        self._stderr_tail: collections.deque[str] = collections.deque(maxlen=40)
+        self._stderr_task: asyncio.Task | None = None
+
+    async def _drain_stderr(self, stream: asyncio.StreamReader) -> None:
+        # The worker redirects fd1 -> fd2, so model/library chatter lands on
+        # stderr. If we PIPE but never read it, the pipe buffer fills and the
+        # worker blocks. So we drain continuously, keeping only the last lines.
+        try:
+            while True:
+                line = await stream.readline()
+                if not line:
+                    return
+                self._stderr_tail.append(line.decode(errors="replace").rstrip())
+        except asyncio.CancelledError:
+            raise
+        except Exception:  # draining must never crash the caller
+            return
+
+    def _stderr_hint(self) -> str:
+        tail = "\n".join(self._stderr_tail)
+        return f" worker stderr tail:\n{tail}" if tail else " (worker produced no stderr)"
+
+    async def warmup(self) -> None:
+        """Load the model now so the first real utterance is transcribed warm."""
+        await self._ensure()
+
+    async def _ensure(self) -> None:
+        if self._proc is not None and self._proc.returncode is None:
+            return
+        env = {
+            **os.environ,
+            "WSAI_WHISPER_MODEL": self.model,
+            "WSAI_WHISPER_DEVICE": self.device,
+        }
+        # ctranslate2 dlopens libcublas/libcudnn from the whisper venv's nvidia
+        # pip packages; the dynamic loader only honours LD_LIBRARY_PATH captured
+        # at exec, so inject those lib dirs into the child env here (harmless on
+        # CPU). Without this the CUDA model loads but transcribe() dies with
+        # "Library libcublas.so.12 is not found".
+        lib_dirs = _cuda_lib_dirs(self.python)
+        if lib_dirs:
+            prev = env.get("LD_LIBRARY_PATH", "")
+            env["LD_LIBRARY_PATH"] = ":".join(lib_dirs + ([prev] if prev else []))
+        if self.language:
+            env["WSAI_WHISPER_LANGUAGE"] = self.language
+        repo_root = str(Path(__file__).resolve().parents[2])
+        self._proc = await asyncio.create_subprocess_exec(
+            self.python, "-m", "wsai.backends.whisper_worker",
+            cwd=repo_root, env=env,
+            stdin=asyncio.subprocess.PIPE,
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+        )
+        self._stderr_tail.clear()
+        assert self._proc.stderr is not None
+        self._stderr_task = asyncio.create_task(self._drain_stderr(self._proc.stderr))
+        ready = await self._proc.stdout.readline()
+        if not ready:  # worker died before signalling ready
+            await self._proc.wait()
+            raise RuntimeError(
+                f"whisper worker exited before ready (code {self._proc.returncode})."
+                f"{self._stderr_hint()}"
+            )
+        try:
+            info = json.loads(ready.decode())
+        except json.JSONDecodeError as exc:
+            raise RuntimeError(
+                f"whisper worker sent invalid ready line {ready!r}: {exc}."
+                f"{self._stderr_hint()}"
+            ) from exc
+        if not info.get("ready"):
+            raise RuntimeError(
+                f"whisper worker failed to start: {info}.{self._stderr_hint()}"
+            )
+        self.load_ms = info.get("ms")
+        self.resolved_device = info.get("device")
+        log.info(
+            "whisper worker ready in %s ms on %s (model %s)",
+            self.load_ms, info.get("device"), info.get("model"),
+        )
+
+    async def transcribe(self, wav_path: str, *, language: str | None = None) -> str:
+        """Transcribe one wav file to text using the warm worker."""
+        await self._ensure()
+        req: dict[str, object] = {"wav": wav_path}
+        lang = language if language is not None else self.language
+        if lang:
+            req["language"] = lang
+        s = time.monotonic()
+        async with self._lock:
+            assert self._proc and self._proc.stdin and self._proc.stdout
+            self._proc.stdin.write((json.dumps(req) + "\n").encode())
+            await self._proc.stdin.drain()
+            resp = await self._proc.stdout.readline()
+        if not resp:
+            raise RuntimeError(f"whisper worker closed unexpectedly.{self._stderr_hint()}")
+        res = json.loads(resp.decode())
+        if not res.get("ok"):
+            raise RuntimeError(f"whisper transcribe failed: {res.get('error')}")
+        log.debug(
+            "transcribe %d ms (worker %s ms): %s",
+            int((time.monotonic() - s) * 1000), res.get("ms"), res.get("text", "")[:60],
+        )
+        return res.get("text", "")
+
+    async def utterances(self) -> AsyncIterator[Utterance]:
+        """Yield an Utterance per finished-utterance wav from `audio_source`.
+
+        With no audio source wired yet (Discord voice receiver pending) this
+        idles and returns, leaving the voice loop dormant but valid."""
+        if self.audio_source is None:
+            return
+        async for wav_path in self.audio_source:
+            text = await self.transcribe(wav_path)
+            if text:
+                yield Utterance(text=text, ts=time.monotonic(), source="voice")
+
+    async def aclose(self) -> None:
+        if self._proc is not None and self._proc.returncode is None:
+            try:
+                self._proc.terminate()
+                await asyncio.wait_for(self._proc.wait(), timeout=5)
+            except (ProcessLookupError, asyncio.TimeoutError):
+                pass
+        if self._stderr_task is not None:
+            self._stderr_task.cancel()
+            try:
+                await self._stderr_task
+            except (asyncio.CancelledError, Exception):
+                pass
+            self._stderr_task = None
+        self._proc = None

wsai/backends/whisper_worker.py

@@ -0,0 +1,122 @@
+"""Persistent faster-whisper STT worker.
+
+faster-whisper (ctranslate2) lives in its own Python (whisper312); loading the
+model takes seconds, so we load it ONCE here and then serve transcription
+requests over stdin/stdout. This process is launched with the whisper312
+interpreter by wsai.backends.whisper.WhisperSTT.
+
+Like the MeloTTS worker, model/backend chatter could corrupt the JSON protocol,
+so on startup we split the streams: a private duplicate of the original stdout
+carries the protocol, and fd 1 is redirected to fd 2 so any library print lands
+on stderr instead (where the parent drains it for diagnostics).
+
+Protocol (one JSON object per line, on the protocol channel):
+    <- {"wav": "/abs/path.wav", "language": "ko"}
+    -> {"ok": true, "text": "...", "language": "ko", "ms": 123}
+    -> {"ok": false, "error": "..."}
+On startup, once the model is ready, it emits exactly one line:
+    -> {"ready": true, "ms": <load-ms>, "device": "cpu", "model": "small"}
+"""
+
+import json
+import os
+import sys
+import time
+
+# Split protocol from library noise BEFORE importing anything heavy.
+_proto = os.fdopen(os.dup(1), "w", buffering=1)  # private copy of real stdout
+os.dup2(2, 1)  # fd1 -> stderr, so stray library prints don't hit the protocol
+
+
+def _emit(obj: dict) -> None:
+    _proto.write(json.dumps(obj, ensure_ascii=False) + "\n")
+    _proto.flush()
+
+
+def _log(*a):
+    print(*a, file=sys.stderr, flush=True)
+
+
+def main() -> None:
+    model_name = os.environ.get("WSAI_WHISPER_MODEL", "small")
+    requested = os.environ.get("WSAI_WHISPER_DEVICE", "auto")  # cpu | cuda | auto
+    default_lang = os.environ.get("WSAI_WHISPER_LANGUAGE", "ko") or None
+
+    from faster_whisper import WhisperModel  # heavy import; only in whisper venv
+
+    def _has_cuda() -> bool:
+        try:
+            import ctranslate2
+
+            return ctranslate2.get_cuda_device_count() > 0
+        except Exception:
+            return False
+
+    device = requested
+    if requested == "auto":
+        device = "cuda" if _has_cuda() else "cpu"
+
+    def _compute_for(dev: str) -> str:
+        # int8 on CPU keeps a small model fast; float16 is the usual CUDA choice.
+        return os.environ.get(
+            "WSAI_WHISPER_COMPUTE", "int8" if dev == "cpu" else "float16"
+        )
+
+    t0 = time.monotonic()
+    try:
+        model = WhisperModel(model_name, device=device, compute_type=_compute_for(device))
+    except Exception as exc:
+        # CUDA picked but unusable (missing libs, OOM): fall back to CPU rather
+        # than leaving the whole voice loop dead.
+        if device == "cuda":
+            _log(f"[whisper_worker] CUDA load failed ({exc}); falling back to CPU")
+            device = "cpu"
+            model = WhisperModel(model_name, device=device, compute_type=_compute_for(device))
+        else:
+            raise
+    compute = _compute_for(device)
+    load_ms = int((time.monotonic() - t0) * 1000)
+
+    # Warm up before signalling ready: the first CUDA transcribe pays a large
+    # lazy cost (kernel autotune), which would slow the first real utterance.
+    # Run a dummy transcribe on 1s of silence here so "ready" means "hot".
+    warmup_ms = None
+    try:
+        import numpy as np
+
+        w = time.monotonic()
+        segs, _ = model.transcribe(np.zeros(16000, dtype=np.float32), language=default_lang)
+        for _ in segs:  # segments are lazy; drain to force the actual compute
+            pass
+        warmup_ms = int((time.monotonic() - w) * 1000)
+    except Exception as exc:
+        _log(f"[whisper_worker] warmup skipped: {exc}")
+
+    _emit({"ready": True, "ms": load_ms, "device": device, "model": model_name, "warmup_ms": warmup_ms})
+    _log(f"[whisper_worker] {model_name} ready in {load_ms} ms on {device}/{compute} (warmup {warmup_ms} ms)")
+
+    for line in sys.stdin:
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            req = json.loads(line)
+            wav = req["wav"]
+            language = req.get("language", default_lang)
+            s = time.monotonic()
+            segments, info = model.transcribe(
+                wav,
+                language=language,
+                beam_size=int(req.get("beam_size", 5)),
+                vad_filter=bool(req.get("vad_filter", True)),
+            )
+            text = "".join(seg.text for seg in segments).strip()
+            ms = int((time.monotonic() - s) * 1000)
+            _emit({"ok": True, "text": text, "language": info.language, "ms": ms})
+        except Exception as exc:  # keep the worker alive across bad requests
+            _emit({"ok": False, "error": f"{type(exc).__name__}: {exc}"})
+            _log(f"[whisper_worker] error: {exc}")
+
+
+if __name__ == "__main__":
+    main()

wsai/config.py

@@ -0,0 +1,62 @@
+"""Configuration. Each field names a backend; the factory maps names -> classes.
+
+Defaults are all "mock" so the skeleton runs out of the box. Flip individual
+fields (via env or code) as real backends land.
+
+Env overrides (optional):
+    WSAI_SOURCE, WSAI_VISION, WSAI_STT, WSAI_TTS, WSAI_BRAIN, WSAI_TEXT
+    WSAI_CAPTURE_INTERVAL
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+
+@dataclass
+class Settings:
+    source: str | None = "mock"  # mock | mss | None (eyes-free)
+    vision: str | None = "mock"  # mock | claude | None (eyes-free)
+    stt: str | None = "mock"     # mock | whisper   | None
+    tts: str | None = "mock"     # mock | melo      | None
+    brain: str = "mock"          # mock | claude
+    text: str | None = None      # None | (discord)
+
+    capture_interval: float = 1.5
+    anthropic_model: str = "claude-sonnet-4-5"
+
+    @classmethod
+    def from_env(cls) -> "Settings":
+        def opt(name: str, default):
+            v = os.environ.get(name)
+            return default if v is None else (None if v.lower() == "none" else v)
+
+        return cls(
+            source=opt("WSAI_SOURCE", "mock"),
+            vision=opt("WSAI_VISION", "mock"),
+            stt=opt("WSAI_STT", "mock"),
+            tts=opt("WSAI_TTS", "mock"),
+            brain=opt("WSAI_BRAIN", "mock"),
+            text=opt("WSAI_TEXT", None),
+            capture_interval=float(os.environ.get("WSAI_CAPTURE_INTERVAL", "1.5")),
+        )
+
+    @classmethod
+    def mock(cls) -> "Settings":
+        return cls()
+
+    @classmethod
+    def live(cls) -> "Settings":
+        """A realistic local config: capture this screen, Claude eyes+brain,
+        mock voice (until STT/TTS backends are wired)."""
+        return cls(source="mss", vision="claude", brain="claude", stt="mock", tts="mock")
+
+    @classmethod
+    def voice(cls) -> "Settings":
+        """Eyes-free voice loop: no screen share, just STT -> Brain -> TTS.
+
+        Screen capture is deferred, so source/vision are off. Backends default
+        to mock so it runs out of the box; flip stt/tts/brain to real ones as
+        they land."""
+        return cls(source=None, vision=None, stt="mock", tts="mock", brain="mock")

wsai/dashboard.py

@@ -0,0 +1,674 @@
+"""Live status website for the voice loop.
+
+Serves a single self-contained page plus a Server-Sent-Events stream so you can
+open a browser and watch, step by step: is it listening, what it heard, what it
+thought/answered, how long each stage took, and whether anything errored.
+
+Pure stdlib (``http.server``). Runs in a background thread so it never blocks
+the asyncio pipeline.
+
+Endpoints:
+    GET /            -> the dashboard HTML
+    GET /api/state   -> JSON snapshot (initial load / fallback polling)
+    GET /events      -> text/event-stream live push
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import queue
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+from .monitor import Monitor
+
+log = logging.getLogger("wsai.dashboard")
+
+def _speech_text(reply: str) -> str:
+    """Return reply text exactly as authored for the TTS backend.
+
+    The TTS backend itself understands bracketed emotion tags: known emotion
+    tags steer delivery and are not spoken; non-emotion brackets are spoken.
+    Do not rewrite a leading tag here, or the first emotion would be read aloud
+    and lost before ``MeloTTS.synth`` can parse it."""
+    return reply
+
+
+def _make_handler(dash: "Dashboard"):
+    monitor = dash.monitor
+
+    class Handler(BaseHTTPRequestHandler):
+        # Quiet: don't spam the console with one line per request.
+        def log_message(self, *args) -> None:  # noqa: D401
+            return
+
+        def _send(self, code: int, body: bytes, ctype: str) -> None:
+            self.send_response(code)
+            self.send_header("Content-Type", ctype)
+            self.send_header("Content-Length", str(len(body)))
+            self.send_header("Cache-Control", "no-store")
+            self.end_headers()
+            self.wfile.write(body)
+
+        def do_GET(self) -> None:  # noqa: N802
+            path = self.path.split("?", 1)[0]
+            if path == "/" or path == "/index.html":
+                self._send(200, PAGE.encode("utf-8"), "text/html; charset=utf-8")
+            elif path == "/api/state":
+                body = json.dumps(monitor.snapshot(), ensure_ascii=False).encode("utf-8")
+                self._send(200, body, "application/json; charset=utf-8")
+            elif path == "/events":
+                self._stream_events()
+            else:
+                self._send(404, b"not found", "text/plain; charset=utf-8")
+
+        def do_POST(self) -> None:  # noqa: N802
+            path = self.path.split("?", 1)[0]
+            if path == "/api/stt":
+                self._handle_stt()
+            elif path == "/api/voice-turn":
+                self._handle_voice_turn()
+            else:
+                self._send(404, b"not found", "text/plain; charset=utf-8")
+
+        def _read_body(self) -> bytes:
+            try:
+                length = int(self.headers.get("Content-Length", "0"))
+            except ValueError:
+                length = 0
+            return self.rfile.read(length) if length > 0 else b""
+
+        def _handle_voice_turn(self) -> None:
+            """Discord voice bridge: utterance wav in -> reply wav out. The
+            recognised/reply text ride along as URL-encoded response headers so
+            the bot can log them; the body is the reply audio to play back."""
+            import urllib.parse
+
+            if dash.stt is None or dash.tts is None:
+                self._send(503, json.dumps({"ok": False, "error": "voice loop not enabled"}).encode(),
+                           "application/json; charset=utf-8")
+                return
+            raw = self._read_body()
+            if not raw:
+                self._send(400, json.dumps({"ok": False, "error": "empty upload"}).encode(),
+                           "application/json; charset=utf-8")
+                return
+            try:
+                res = dash.voice_turn(raw)
+            except Exception as exc:  # noqa: BLE001
+                log.exception("voice-turn failed")
+                self._send(500, json.dumps({"ok": False, "error": f"{type(exc).__name__}: {exc}"},
+                                           ensure_ascii=False).encode(), "application/json; charset=utf-8")
+                return
+            body = res["wav"]
+            self.send_response(200 if body else 204)
+            self.send_header("Content-Type", "audio/wav")
+            self.send_header("Content-Length", str(len(body)))
+            self.send_header("X-Heard", urllib.parse.quote(res.get("heard", "")))
+            self.send_header("X-Reply", urllib.parse.quote(res.get("reply", "")))
+            self.send_header("Cache-Control", "no-store")
+            self.end_headers()
+            if body:
+                self.wfile.write(body)
+
+        def _handle_stt(self) -> None:
+            """Accept an uploaded audio blob (mic recording or file), run it
+            through the real GPU STT, and return the recognised text."""
+            if dash.stt is None:
+                self._send(503, json.dumps({"ok": False, "error": "STT not enabled"}).encode(),
+                           "application/json; charset=utf-8")
+                return
+            try:
+                length = int(self.headers.get("Content-Length", "0"))
+            except ValueError:
+                length = 0
+            if length <= 0:
+                self._send(400, json.dumps({"ok": False, "error": "empty upload"}).encode(),
+                           "application/json; charset=utf-8")
+                return
+            raw = self.rfile.read(length)
+            try:
+                result = dash.transcribe_upload(raw)
+                body = json.dumps({"ok": True, **result}, ensure_ascii=False).encode("utf-8")
+                self._send(200, body, "application/json; charset=utf-8")
+            except Exception as exc:  # noqa: BLE001 — surface the reason to the page
+                log.exception("STT upload failed")
+                body = json.dumps({"ok": False, "error": f"{type(exc).__name__}: {exc}"},
+                                  ensure_ascii=False).encode("utf-8")
+                self._send(500, body, "application/json; charset=utf-8")
+
+        def _stream_events(self) -> None:
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream; charset=utf-8")
+            self.send_header("Cache-Control", "no-store")
+            self.send_header("Connection", "keep-alive")
+            self.end_headers()
+            q = monitor.subscribe()
+            try:
+                # Prime the client with a full snapshot so it renders instantly.
+                first = json.dumps(
+                    {"type": "snapshot", "snapshot": monitor.snapshot()},
+                    ensure_ascii=False,
+                )
+                self.wfile.write(f"data: {first}\n\n".encode("utf-8"))
+                self.wfile.flush()
+                while True:
+                    try:
+                        data = q.get(timeout=15)
+                    except queue.Empty:
+                        # Heartbeat keeps proxies / the browser from timing out.
+                        self.wfile.write(b": ping\n\n")
+                        self.wfile.flush()
+                        continue
+                    self.wfile.write(f"data: {data}\n\n".encode("utf-8"))
+                    self.wfile.flush()
+            except (BrokenPipeError, ConnectionResetError):
+                pass
+            finally:
+                monitor.unsubscribe(q)
+
+    return Handler
+
+
+class Dashboard:
+    """Owns the HTTP server thread.
+
+    Optionally holds a real STT backend so the page can offer a live
+    recognition test (upload/record audio -> GPU whisper -> text). The STT
+    backend is async, so the dashboard runs its own asyncio loop in a
+    background thread and bridges the synchronous HTTP handlers onto it.
+    """
+
+    def __init__(self, monitor: Monitor, host: str = "0.0.0.0", port: int = 8787,
+                 stt=None, tts=None, brain=None, history_turns: int = 12) -> None:
+        self.monitor = monitor
+        self.host = host
+        self.port = port
+        self.stt = stt
+        self.tts = tts
+        self.brain = brain
+        self._history: list[tuple[str, str]] = []
+        self._history_turns = history_turns
+        self._server: ThreadingHTTPServer | None = None
+        self._thread: threading.Thread | None = None
+        self._loop = None
+        self._loop_thread: threading.Thread | None = None
+
+    def start(self) -> None:
+        if self.stt is not None or self.tts is not None:
+            self._start_loop()
+        handler = _make_handler(self)
+        self._server = ThreadingHTTPServer((self.host, self.port), handler)
+        self._server.daemon_threads = True
+        self._thread = threading.Thread(
+            target=self._server.serve_forever, name="wsai-dashboard", daemon=True
+        )
+        self._thread.start()
+        log.info("dashboard on http://%s:%d", self.host, self.port)
+
+    def stop(self) -> None:
+        if self._server is not None:
+            self._server.shutdown()
+            self._server.server_close()
+            self._server = None
+        if self._loop is not None:
+            self._loop.call_soon_threadsafe(self._loop.stop)
+            self._loop = None
+
+    # -- async bridge (STT test) ----------------------------------------- #
+    def _start_loop(self) -> None:
+        import asyncio
+
+        self._loop = asyncio.new_event_loop()
+        self._loop_thread = threading.Thread(
+            target=self._loop.run_forever, name="wsai-dashboard-loop", daemon=True
+        )
+        self._loop_thread.start()
+
+    def _submit(self, coro, timeout: float = 120.0):
+        import asyncio
+
+        fut = asyncio.run_coroutine_threadsafe(coro, self._loop)
+        return fut.result(timeout=timeout)
+
+    def warm(self) -> None:
+        """Pre-start the STT/TTS workers (loads + warms the GPU) so the first
+        recognition/synth is instant instead of paying model-load + CUDA autotune."""
+        if self.stt is not None:
+            self._submit(self.stt._ensure())
+        if self.tts is not None:
+            self._submit(self.tts._ensure())
+
+    def voice_turn(self, audio_bytes: bytes) -> dict:
+        """One Discord voice turn: decode the uploaded utterance, recognise it
+        on the GPU, think of a reply (Claude brain if wired, else echo),
+        synthesise it on the GPU, and return {heard, reply, wav} where wav is the
+        reply audio bytes for the bot to play back into the channel."""
+        import os
+        import subprocess
+        import time
+        import uuid
+
+        if self.stt is None or self.tts is None:
+            raise RuntimeError("voice_turn needs both STT and TTS")
+        updir = os.path.expanduser("~/.cache/wsai/uploads")
+        os.makedirs(updir, exist_ok=True)
+        stem = os.path.join(updir, uuid.uuid4().hex)
+        src, wav = stem + ".bin", stem + ".wav"
+        with open(src, "wb") as f:
+            f.write(audio_bytes)
+        turn = self.monitor.turn(source="discord")
+        t0 = time.monotonic()
+        try:
+            subprocess.run(
+                ["ffmpeg", "-y", "-i", src, "-ar", "16000", "-ac", "1", wav],
+                check=True, capture_output=True,
+            )
+            heard = (self._submit(self.stt.transcribe(wav)) or "").strip()
+            turn.heard(heard or "(빈 결과)")
+            if not heard:
+                # Nothing recognised (silence/noise): mark it as [잡음] and skip
+                # the brain/TTS so the bot plays nothing back.
+                turn.replied("[잡음]")
+                turn.finish()
+                return {"heard": heard, "reply": "[잡음]", "wav": b""}
+            reply_text = self._think(heard)
+            turn.replied(reply_text)
+            out_path = self._submit(self.tts.synth(_speech_text(reply_text)))
+            with open(out_path, "rb") as f:
+                reply_wav = f.read()
+            ms = int((time.monotonic() - t0) * 1000)
+            step = turn.step("STT+두뇌+TTS" if self.brain else "STT+TTS(GPU)")
+            step.ok, step.ms = True, float(ms)
+            turn._steps.append(step)
+            turn.finish()
+            try:
+                os.remove(out_path)
+            except OSError:
+                pass
+            return {"heard": heard, "reply": reply_text, "wav": reply_wav}
+        except subprocess.CalledProcessError as exc:
+            turn.finish(error="ffmpeg decode failed")
+            err = exc.stderr.decode("utf-8", "replace")[-300:] if exc.stderr else str(exc)
+            raise RuntimeError(f"ffmpeg: {err}") from exc
+        except Exception as exc:
+            turn.finish(error=str(exc))
+            raise
+        finally:
+            for p in (src, wav):
+                try:
+                    os.remove(p)
+                except OSError:
+                    pass
+
+    def _think(self, heard: str) -> str:
+        """Turn what was heard into a reply. Uses the Claude brain when wired
+        (with rolling conversation history); falls back to echo if there is no
+        brain, and to a spoken apology if the brain call fails — so one API hiccup
+        never kills the voice loop."""
+        if self.brain is None:
+            return heard  # echo mode
+        try:
+            reply = self._submit(self.brain.respond(heard, None, list(self._history)))
+            text = (reply.text or "").strip()
+        except Exception as exc:  # noqa: BLE001
+            log.exception("brain failed")
+            self.monitor.log("error", f"두뇌 응답 실패: {exc}")
+            blob = f"{getattr(exc, 'status_code', '')} {exc}".lower()
+            if "529" in blob or "overload" in blob:
+                # Transient server overload survived the SDK retries.
+                return "지금 서버가 잠깐 붐벼서 생각이 늦네. 잠시 뒤에 다시 말해줄래?"
+            return "미안, 지금 잠깐 생각이 안 났어. 다시 말해줄래?"
+        if not text:
+            return "음, 뭐라고 해야 할지 모르겠어. 다시 말해줄래?"
+        self._history.append((heard, text))
+        if len(self._history) > self._history_turns:
+            self._history = self._history[-self._history_turns:]
+        return text
+
+    def transcribe_upload(self, audio_bytes: bytes) -> dict:
+        """ffmpeg-normalise an uploaded blob to 16 kHz mono wav, transcribe it
+        on the GPU, and record the result as a monitor turn so it also shows in
+        the live feed. Returns {text, ms, device}."""
+        import os
+        import subprocess
+        import tempfile
+        import time
+        import uuid
+
+        updir = os.path.expanduser("~/.cache/wsai/uploads")
+        os.makedirs(updir, exist_ok=True)
+        stem = os.path.join(updir, uuid.uuid4().hex)
+        src, wav = stem + ".bin", stem + ".wav"
+        with open(src, "wb") as f:
+            f.write(audio_bytes)
+        turn = self.monitor.turn(source="web")
+        t0 = time.monotonic()
+        try:
+            # Decode whatever the browser sent (webm/opus, ogg, mp4, wav) to the
+            # 16 kHz mono wav faster-whisper expects.
+            subprocess.run(
+                ["ffmpeg", "-y", "-i", src, "-ar", "16000", "-ac", "1", wav],
+                check=True, capture_output=True,
+            )
+            text = self._submit(self.stt.transcribe(wav))
+            ms = int((time.monotonic() - t0) * 1000)
+            turn.heard(text or "(빈 결과)")
+            step = turn.step("STT(GPU)")
+            step.ok, step.ms = True, float(ms)
+            turn._steps.append(step)
+            turn.finish()
+            return {"text": text, "ms": ms,
+                    "device": getattr(self.stt, "resolved_device", None) or "?"}
+        except subprocess.CalledProcessError as exc:
+            turn.finish(error="ffmpeg decode failed")
+            err = exc.stderr.decode("utf-8", "replace")[-300:] if exc.stderr else str(exc)
+            raise RuntimeError(f"ffmpeg: {err}") from exc
+        except Exception as exc:
+            turn.finish(error=str(exc))
+            raise
+        finally:
+            for p in (src, wav):
+                try:
+                    os.remove(p)
+                except OSError:
+                    pass
+
+
+# --------------------------------------------------------------------------- #
+# The page. One file, no external assets, so it works offline / behind a LAN.
+# --------------------------------------------------------------------------- #
+PAGE = r"""<!DOCTYPE html>
+<html lang="ko">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>watch_sceen_ai · 실시간 상태</title>
+<style>
+  :root{
+    --bg:#0b0f14; --panel:#131a22; --panel2:#0f151c; --line:#223040;
+    --fg:#e6edf3; --muted:#8aa0b2; --accent:#3fb6ff; --ok:#37d67a;
+    --err:#ff5c6c; --warn:#ffc857; --heard:#7aa2ff; --reply:#b28bff;
+  }
+  *{box-sizing:border-box}
+  body{margin:0;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Apple SD Gothic Neo","Malgun Gothic",sans-serif;
+    background:var(--bg);color:var(--fg);line-height:1.5}
+  header{position:sticky;top:0;z-index:5;background:linear-gradient(180deg,#0d141c,#0b0f14);
+    border-bottom:1px solid var(--line);padding:14px 20px;display:flex;flex-wrap:wrap;gap:16px;align-items:center}
+  h1{font-size:16px;margin:0;font-weight:650;letter-spacing:.2px}
+  .sub{color:var(--muted);font-size:12px}
+  .pill{display:inline-flex;align-items:center;gap:7px;padding:5px 11px;border-radius:999px;
+    background:var(--panel);border:1px solid var(--line);font-size:12.5px;color:var(--muted)}
+  .dot{width:9px;height:9px;border-radius:50%;background:#556}
+  .dot.live{background:var(--ok);box-shadow:0 0 0 0 rgba(55,214,122,.6);animation:pulse 1.6s infinite}
+  .dot.off{background:#556}
+  @keyframes pulse{0%{box-shadow:0 0 0 0 rgba(55,214,122,.55)}70%{box-shadow:0 0 0 9px rgba(55,214,122,0)}100%{box-shadow:0 0 0 0 rgba(55,214,122,0)}}
+  .stats{display:flex;gap:10px;flex-wrap:wrap;margin-left:auto}
+  .stat{background:var(--panel);border:1px solid var(--line);border-radius:10px;padding:6px 12px;min-width:78px}
+  .stat b{display:block;font-size:16px}
+  .stat span{color:var(--muted);font-size:11px}
+  main{max-width:1000px;margin:0 auto;padding:18px 20px 60px}
+  .comp{display:flex;gap:8px;flex-wrap:wrap;margin:2px 0 18px}
+  .comp .pill{font-size:11.5px}
+  .turn{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:14px 16px;margin:12px 0;
+    animation:rise .25s ease}
+  @keyframes rise{from{opacity:0;transform:translateY(6px)}to{opacity:1;transform:none}}
+  .turn.active{border-color:#2b5d86;box-shadow:0 0 0 1px #14324a inset}
+  .turn.error{border-color:#5c2530}
+  .trow{display:flex;align-items:baseline;gap:10px;margin-bottom:8px}
+  .badge{font-size:11px;padding:2px 9px;border-radius:999px;border:1px solid var(--line);color:var(--muted);white-space:nowrap}
+  .badge.ok{color:var(--ok);border-color:#1f5236}
+  .badge.error{color:var(--err);border-color:#5c2530}
+  .badge.active{color:var(--accent);border-color:#234a63}
+  .time{color:var(--muted);font-size:11.5px;margin-left:auto}
+  .line{display:flex;gap:9px;margin:5px 0;align-items:flex-start}
+  .tag{flex:0 0 42px;font-size:11px;color:var(--muted);padding-top:2px}
+  .heard{color:var(--heard);font-weight:550}
+  .reply{color:var(--reply);font-weight:550}
+  .steps{margin-top:10px;border-top:1px dashed var(--line);padding-top:10px;display:flex;flex-direction:column;gap:6px}
+  .step{display:grid;grid-template-columns:120px 1fr 66px;gap:10px;align-items:center;font-size:12.5px}
+  .step .sname{color:var(--muted)}
+  .step .sbar{height:8px;background:var(--panel2);border-radius:6px;overflow:hidden;border:1px solid var(--line)}
+  .step .sfill{height:100%;background:linear-gradient(90deg,#2b7bb0,#3fb6ff)}
+  .step.err .sfill{background:linear-gradient(90deg,#7a2531,#ff5c6c)}
+  .step .sms{text-align:right;color:var(--fg);font-variant-numeric:tabular-nums}
+  .step .serr{grid-column:1 / -1;color:var(--err);font-size:11.5px}
+  .total{margin-top:8px;font-size:12px;color:var(--muted)}
+  .total b{color:var(--fg)}
+  .empty{color:var(--muted);text-align:center;padding:50px 0;font-size:14px}
+  .events{margin-top:26px}
+  .events h2{font-size:13px;color:var(--muted);font-weight:600;margin:0 0 8px}
+  .ev{font-size:12px;color:var(--muted);padding:3px 0;border-bottom:1px solid #16202b;display:flex;gap:10px}
+  .ev.error{color:var(--err)}
+  .ev .et{flex:0 0 68px;color:#5f7488}
+  code{background:#0c1219;padding:1px 5px;border-radius:5px;border:1px solid var(--line)}
+  .demobar{background:#2a2210;border:1px solid #6b5417;color:var(--warn);border-radius:12px;
+    padding:11px 15px;margin:0 0 16px;font-size:13px;line-height:1.55}
+  .demobar b{color:#ffe08a}
+  .sttbox{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:14px 16px;margin:0 0 16px}
+  .sttbox h2{font-size:13px;margin:0 0 10px;font-weight:600;color:var(--fg)}
+  .sttrow{display:flex;gap:10px;align-items:center;flex-wrap:wrap}
+  .btn{background:#173042;border:1px solid #234a63;color:var(--fg);border-radius:10px;padding:8px 14px;font-size:13px;cursor:pointer}
+  .btn:hover{background:#1d3d54}
+  .btn.rec{background:#4a1f27;border-color:#7a2531;color:#ffb3bb}
+  .sttstat{color:var(--muted);font-size:12.5px}
+  .sttres{margin-top:12px;font-size:15px;min-height:1px}
+  .sttres .txt{color:var(--heard);font-weight:600;line-height:1.5}
+  .sttres .meta{color:var(--muted);font-size:12px;margin-top:5px}
+</style>
+</head>
+<body>
+<header>
+  <div>
+    <h1>watch_sceen_ai · 실시간 상태</h1>
+    <div class="sub">STT → 두뇌 → TTS 음성 루프를 단계별로 관찰</div>
+  </div>
+  <div class="pill"><span id="dot" class="dot off"></span><span id="listen">연결 대기</span></div>
+  <div class="stats">
+    <div class="stat"><b id="s-turns">0</b><span>대화 수</span></div>
+    <div class="stat"><b id="s-errors">0</b><span>오류</span></div>
+    <div class="stat"><b id="s-up">0초</b><span>가동시간</span></div>
+    <div class="stat"><b id="s-conn">·</b><span>연결</span></div>
+  </div>
+</header>
+<main>
+  <section class="sttbox" id="sttbox" style="display:none">
+    <h2>🎤 음성 인식(STT) 테스트 · GPU</h2>
+    <div class="sttrow">
+      <button id="recbtn" class="btn">🎤 녹음 시작</button>
+      <label class="btn" for="fileinp">📁 오디오 파일 올리기</label>
+      <input id="fileinp" type="file" accept="audio/*" hidden>
+      <span id="ststat" class="sttstat">녹음하거나 오디오 파일을 올리면 GPU로 인식합니다.</span>
+    </div>
+    <div id="sttres" class="sttres"></div>
+  </section>
+  <div class="demobar" id="demobar" style="display:none"></div>
+  <div class="comp" id="comp"></div>
+  <div id="turns"></div>
+  <div id="empty" class="empty">아직 대화가 없습니다. 사용자가 말하면 여기에 단계별로 나타납니다.</div>
+  <div class="events">
+    <h2>이벤트 / 오류 로그</h2>
+    <div id="events"></div>
+  </div>
+</main>
+<script>
+const $ = (id)=>document.getElementById(id);
+const turns = new Map();   // id -> turn object
+let statusData = null;
+
+function fmtTime(wall){
+  const d = new Date(wall*1000);
+  return d.toLocaleTimeString('ko-KR',{hour12:false}) +
+    '.' + String(d.getMilliseconds()).padStart(3,'0');
+}
+function fmtUptime(s){
+  s = Math.floor(s||0);
+  const h=Math.floor(s/3600), m=Math.floor(s%3600/60), sec=s%60;
+  if(h) return h+'시간 '+m+'분';
+  if(m) return m+'분 '+sec+'초';
+  return sec+'초';
+}
+function esc(t){const d=document.createElement('div');d.textContent=t==null?'':t;return d.innerHTML;}
+
+function renderStatus(s){
+  statusData = s;
+  $('s-turns').textContent = s.turns_total ?? 0;
+  $('s-errors').textContent = s.errors_total ?? 0;
+  $('s-up').textContent = fmtUptime(s.uptime_s);
+  const listening = s.listening;
+  $('dot').className = 'dot ' + (listening ? 'live' : 'off');
+  $('listen').textContent = listening ? '듣는 중' : (s.running ? '실행 중 (대기)' : '중지됨');
+  const comps = s.components || {};
+  // Demo banner: if the ears/brain/mouth are still mock, everything below is
+  // replayed sample data, not a real conversation. Say so loudly.
+  const sttReal = comps.stt && comps.stt!=='mock' && comps.stt!=='none';
+  $('sttbox').style.display = sttReal ? 'block' : 'none';
+  const mockParts = ['stt','brain','tts'].filter(k => comps[k]==='mock');
+  const bar = $('demobar');
+  if(mockParts.length){
+    bar.style.display='block';
+    bar.innerHTML = '⚠ <b>데모 모드</b> — 실제 음성/STT/두뇌/TTS가 아직 연결되지 않아, 아래 대화는 '
+      + '실제로 들은 내용이 아니라 <b>목(mock) 예시 스크립트</b>입니다. '
+      + '실제 엔진(faster-whisper·Claude·MeloTTS)을 붙이면 이 자리에 진짜 발화·지연·오류가 표시됩니다.';
+  } else {
+    bar.style.display='none';
+  }
+  const el = $('comp'); el.innerHTML = '';
+  const names = {source:'눈(소스)', vision:'시각', stt:'귀(STT)', brain:'두뇌', tts:'입(TTS)', text:'텍스트'};
+  for(const k of Object.keys(names)){
+    if(!(k in comps)) continue;
+    const v = comps[k];
+    const p = document.createElement('span');
+    p.className = 'pill';
+    p.innerHTML = '<span class="dot '+(v && v!=='none'?'live':'off')+'"></span>'+names[k]+': <b>&nbsp;'+esc(v||'off')+'</b>';
+    el.appendChild(p);
+  }
+}
+function setConn(ok){ $('s-conn').textContent = ok ? '●' : '○'; $('s-conn').style.color = ok ? 'var(--ok)':'var(--err)'; }
+
+function maxMs(steps){ let m=1; for(const s of steps) m=Math.max(m, s.ms||0); return m; }
+
+function turnEl(t){
+  const wrap = document.createElement('div');
+  wrap.className = 'turn ' + (t.status||'active');
+  wrap.id = 'turn-'+t.id;
+  const mx = maxMs(t.steps);
+  let steps = '';
+  for(const s of (t.steps||[])){
+    const pct = Math.max(3, Math.round((s.ms||0)/mx*100));
+    const err = s.ok===false;
+    steps += '<div class="step'+(err?' err':'')+'">'
+      + '<span class="sname">'+esc(s.name)+'</span>'
+      + '<span class="sbar"><span class="sfill" style="width:'+pct+'%"></span></span>'
+      + '<span class="sms">'+ (s.ms!=null? s.ms.toFixed(0)+' ms':'…') +'</span>'
+      + (err && s.error ? '<span class="serr">⚠ '+esc(s.error)+'</span>':'')
+      + '</div>';
+  }
+  const badge = t.status==='ok' ? '<span class="badge ok">정상</span>'
+    : t.status==='error' ? '<span class="badge error">오류</span>'
+    : '<span class="badge active">진행 중…</span>';
+  wrap.innerHTML =
+    '<div class="trow">'+badge
+      +'<span class="badge">#'+t.id+' · '+esc(t.source||'voice')+'</span>'
+      +'<span class="time">'+fmtTime(t.wall)+'</span></div>'
+    +'<div class="line"><span class="tag">들음</span><span class="heard">'+(t.heard?esc(t.heard):'<i style="color:var(--muted)">(수신 대기)</i>')+'</span></div>'
+    +'<div class="line"><span class="tag">답변</span><span class="reply">'+(t.reply?esc(t.reply):'<i style="color:var(--muted)">…생각 중</i>')+'</span></div>'
+    +(t.error?'<div class="line"><span class="tag">오류</span><span style="color:var(--err)">'+esc(t.error)+'</span></div>':'')
+    +'<div class="steps">'+steps+'</div>'
+    +'<div class="total">총 소요 <b>'+(t.total_ms?t.total_ms.toFixed(0)+' ms':'…')+'</b></div>';
+  return wrap;
+}
+
+function upsertTurn(t){
+  turns.set(t.id, t);
+  $('empty').style.display = 'none';
+  const cont = $('turns');
+  const existing = $('turn-'+t.id);
+  const fresh = turnEl(t);
+  if(existing){ existing.replaceWith(fresh); }
+  else { cont.prepend(fresh); }
+}
+
+function addEvent(e){
+  const box = $('events');
+  const row = document.createElement('div');
+  row.className = 'ev ' + (e.level==='error'?'error':'');
+  row.innerHTML = '<span class="et">'+fmtTime(e.wall)+'</span><span>'+esc(e.message)+'</span>';
+  box.prepend(row);
+  while(box.childElementCount>60) box.removeChild(box.lastChild);
+}
+
+function applySnapshot(snap){
+  renderStatus(snap.status);
+  turns.clear(); $('turns').innerHTML='';
+  const list = (snap.turns||[]);
+  for(const t of list) upsertTurn(t);
+  if(list.length===0){ $('empty').style.display='block'; }
+  $('events').innerHTML='';
+  for(const e of (snap.events||[])) addEvent(e);
+}
+
+function connect(){
+  const es = new EventSource('/events');
+  es.onopen = ()=> setConn(true);
+  es.onerror = ()=> setConn(false);
+  es.onmessage = (m)=>{
+    let ev; try{ ev = JSON.parse(m.data); }catch(_){ return; }
+    if(ev.type==='snapshot') applySnapshot(ev.snapshot);
+    else if(ev.type==='status') renderStatus(ev.status);
+    else if(ev.type==='turn') upsertTurn(ev.turn);
+    else if(ev.type==='log') { addEvent(ev); if(statusData){ statusData.errors_total=(statusData.errors_total||0)+(ev.level==='error'?1:0); $('s-errors').textContent=statusData.errors_total; } }
+  };
+}
+// --- STT recognition test (upload / mic record -> GPU whisper) ----------- #
+let mediaRec=null, chunks=[];
+async function sendBlob(blob){
+  $('ststat').textContent='인식 중… (GPU)';
+  $('sttres').innerHTML='';
+  try{
+    const r=await fetch('/api/stt',{method:'POST',
+      headers:{'Content-Type':blob.type||'application/octet-stream'},body:blob});
+    const j=await r.json();
+    if(j.ok){
+      $('sttres').innerHTML='<div class="txt">'+esc(j.text||'(빈 결과)')+'</div>'
+        +'<div class="meta">인식 '+j.ms+' ms · '+esc(j.device)+'</div>';
+      $('ststat').textContent='완료. 다시 녹음하거나 파일을 올릴 수 있습니다.';
+    }else{
+      $('sttres').innerHTML='<div class="meta" style="color:var(--err)">오류: '+esc(j.error)+'</div>';
+      $('ststat').textContent='실패.';
+    }
+  }catch(e){ $('ststat').textContent='요청 실패: '+e; }
+}
+(function(){
+  const fi=$('fileinp'); if(fi) fi.onchange=()=>{ if(fi.files[0]) sendBlob(fi.files[0]); };
+  const rb=$('recbtn'); if(!rb) return;
+  rb.onclick=async()=>{
+    if(mediaRec && mediaRec.state==='recording'){ mediaRec.stop(); return; }
+    if(!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia){
+      $('ststat').textContent='이 주소(원격 http)에서는 브라우저 마이크가 막혀 있습니다. 파일 업로드를 사용하세요.';
+      return;
+    }
+    try{
+      const stream=await navigator.mediaDevices.getUserMedia({audio:true});
+      chunks=[]; mediaRec=new MediaRecorder(stream);
+      mediaRec.ondataavailable=(e)=>{ if(e.data.size) chunks.push(e.data); };
+      mediaRec.onstop=()=>{
+        stream.getTracks().forEach(t=>t.stop());
+        rb.textContent='🎤 녹음 시작'; rb.classList.remove('rec');
+        sendBlob(new Blob(chunks,{type:mediaRec.mimeType||'audio/webm'}));
+      };
+      mediaRec.start();
+      rb.textContent='⏹ 녹음 중지'; rb.classList.add('rec');
+      $('ststat').textContent='녹음 중… 말한 뒤 중지를 누르세요.';
+    }catch(e){ $('ststat').textContent='마이크 접근 실패: '+e; }
+  };
+})();
+
+connect();
+// Refresh uptime label every second from the last known status.
+setInterval(()=>{ if(statusData){ statusData.uptime_s=(statusData.uptime_s||0)+1; $('s-up').textContent=fmtUptime(statusData.uptime_s);} }, 1000);
+</script>
+</body>
+</html>
+"""

wsai/factory.py

@@ -0,0 +1,96 @@
+"""Build a Pipeline from Settings. This is the single place that knows which
+concrete class each config name maps to, so adding a backend = one line here."""
+
+from __future__ import annotations
+
+from .config import Settings
+from .monitor import Monitor
+from .pipeline import Pipeline
+
+
+def build(settings: Settings, monitor: Monitor | None = None) -> Pipeline:
+    pipe = Pipeline(
+        source=_source(settings),
+        vision=_vision(settings),
+        brain=_brain(settings),
+        stt=_stt(settings),
+        tts=_tts(settings),
+        text_channel=_text(settings),
+        monitor=monitor,
+    )
+    if monitor is not None:
+        monitor.set_components(
+            {
+                "source": settings.source or "none",
+                "vision": settings.vision or "none",
+                "stt": settings.stt or "none",
+                "brain": settings.brain,
+                "tts": settings.tts or "none",
+                "text": settings.text or "none",
+            }
+        )
+    return pipe
+
+
+def _source(s: Settings):
+    if s.source in (None, "none"):
+        return None
+    if s.source == "mss":
+        from .backends.capture_mss import MSSFrameSource
+
+        return MSSFrameSource(interval=s.capture_interval)
+    from .backends.mock import MockFrameSource
+
+    return MockFrameSource(interval=s.capture_interval)
+
+
+def _vision(s: Settings):
+    if s.vision in (None, "none"):
+        return None
+    if s.vision == "claude":
+        from .backends.claude import ClaudeVision
+
+        return ClaudeVision(model=s.anthropic_model)
+    from .backends.mock import MockVision
+
+    return MockVision()
+
+
+def _brain(s: Settings):
+    if s.brain == "claude":
+        from .backends.claude import ClaudeBrain
+
+        return ClaudeBrain(model=s.anthropic_model)
+    from .backends.mock import MockBrain
+
+    return MockBrain()
+
+
+def _stt(s: Settings):
+    if s.stt in (None, "none"):
+        return None
+    if s.stt == "whisper":
+        from .backends.whisper import WhisperSTT
+
+        return WhisperSTT()
+    from .backends.mock import MockSTT
+
+    return MockSTT()
+
+
+def _tts(s: Settings):
+    if s.tts in (None, "none"):
+        return None
+    if s.tts == "melo":
+        from .backends.melo import MeloTTS
+
+        return MeloTTS()
+    from .backends.mock import MockTTS
+
+    return MockTTS()
+
+
+def _text(s: Settings):
+    if s.text in (None, "none"):
+        return None
+    raise NotImplementedError("discord text channel backend not implemented yet")

wsai/interfaces.py

@@ -0,0 +1,136 @@
+"""Core data types and component interfaces for the watch-screen AI.
+
+The whole system is a small pipeline:
+
+    FrameSource  --frames-->  VisionBackend  --observations-->  [SharedScreenContext]
+                                                                        |
+    SpeechToText / TextInput  --utterances-->  Brain  <----------------/
+                                                 |
+                                                 v
+                                     TextToSpeech / TextOutput
+
+Every stage is a Protocol so a concrete backend (mock, local GPU, cloud API,
+discord web capture, ...) can be swapped in from config without touching the
+orchestrator.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import AsyncIterator, Protocol, runtime_checkable
+
+
+# --------------------------------------------------------------------------- #
+# Data that flows through the pipeline
+# --------------------------------------------------------------------------- #
+
+@dataclass
+class Frame:
+    """A single captured image of the shared screen."""
+
+    # Raw encoded image bytes (PNG/JPEG). Kept as bytes so any backend can
+    # decode it however it likes and so it is trivial to base64 for a cloud API.
+    data: bytes
+    width: int
+    height: int
+    # Monotonic capture timestamp in seconds.
+    ts: float
+    mime: str = "image/png"
+
+
+@dataclass
+class ScreenObservation:
+    """What the vision backend understood from a Frame."""
+
+    text: str
+    ts: float
+    # Optional structured hints (e.g. detected app, code language, error text).
+    tags: dict[str, str] = field(default_factory=dict)
+
+
+@dataclass
+class Utterance:
+    """Something the user said (voice→text) or typed."""
+
+    text: str
+    ts: float
+    source: str = "voice"  # "voice" | "text"
+
+
+@dataclass
+class Reply:
+    """The AI's response, ready to be spoken and/or shown."""
+
+    text: str
+    ts: float
+
+
+# --------------------------------------------------------------------------- #
+# Component interfaces
+# --------------------------------------------------------------------------- #
+
+@runtime_checkable
+class FrameSource(Protocol):
+    """Produces frames of the shared screen."""
+
+    async def frames(self) -> AsyncIterator[Frame]:
+        """Yield frames until cancelled. Cadence is up to the implementation."""
+        ...
+
+    async def aclose(self) -> None:
+        ...
+
+
+@runtime_checkable
+class VisionBackend(Protocol):
+    """Turns a Frame into a text description of what is on screen."""
+
+    async def describe(self, frame: Frame, hint: str | None = None) -> ScreenObservation:
+        ...
+
+
+@runtime_checkable
+class SpeechToText(Protocol):
+    """Streams user utterances from the microphone (or a mock source)."""
+
+    async def utterances(self) -> AsyncIterator[Utterance]:
+        ...
+
+    async def aclose(self) -> None:
+        ...
+
+
+@runtime_checkable
+class TextToSpeech(Protocol):
+    """Speaks a reply out loud."""
+
+    async def speak(self, reply: Reply) -> None:
+        ...
+
+
+@runtime_checkable
+class Brain(Protocol):
+    """The conversational LLM. Given the latest screen context, the user's
+    message and the running history, produce a reply."""
+
+    async def respond(
+        self,
+        user_text: str,
+        screen: ScreenObservation | None,
+        history: list[tuple[str, str]],
+    ) -> Reply:
+        ...
+
+
+@runtime_checkable
+class TextChannel(Protocol):
+    """Optional text I/O (e.g. a Discord channel) that mirrors the voice loop."""
+
+    async def messages(self) -> AsyncIterator[Utterance]:
+        ...
+
+    async def send(self, reply: Reply) -> None:
+        ...
+
+    async def aclose(self) -> None:
+        ...

wsai/monitor.py

@@ -0,0 +1,235 @@
+"""Telemetry hub for the live status dashboard.
+
+The pipeline is a chain of steps (heard -> screen context -> brain -> speak).
+This module records, for every conversation turn, *what happened at each step*
+and *how long it took*, plus a rolling status header and any errors. The
+dashboard (``wsai/dashboard.py``) reads a snapshot and subscribes for live
+push updates.
+
+Design notes:
+* Pure stdlib, no deps — matches the project's "core has no third-party deps".
+* Thread-safe. The pipeline mutates it from the asyncio loop; the HTTP server
+  reads/subscribes from its own threads. A single lock guards everything.
+* A Monitor with zero subscribers is essentially free, so the pipeline can
+  always hold one (no separate no-op path).
+"""
+
+from __future__ import annotations
+
+import json
+import queue
+import threading
+import time
+from collections import deque
+from typing import Any
+
+
+def _now_wall() -> float:
+    # Wall-clock seconds for human-readable timestamps on the page.
+    return time.time()
+
+
+def _now_mono() -> float:
+    # Monotonic seconds for measuring durations (immune to clock jumps).
+    return time.monotonic()
+
+
+class Step:
+    """One timed stage inside a turn (e.g. "두뇌"). Used as an async context
+    manager so it can wrap an ``await`` and record ok/error + elapsed ms."""
+
+    def __init__(self, turn: "Turn", name: str) -> None:
+        self.turn = turn
+        self.name = name
+        self.ok: bool | None = None
+        self.ms: float = 0.0
+        self.detail: str = ""
+        self.error: str = ""
+        self._t0 = 0.0
+
+    async def __aenter__(self) -> "Step":
+        self._t0 = _now_mono()
+        self.turn._steps.append(self)
+        self.turn._touch()
+        return self
+
+    async def __aexit__(self, exc_type, exc, tb) -> bool:
+        self.ms = (_now_mono() - self._t0) * 1000.0
+        if exc is not None:
+            self.ok = False
+            self.error = f"{exc_type.__name__}: {exc}"
+        else:
+            self.ok = True
+        self.turn._touch()
+        return False  # never swallow: the pipeline/TaskGroup must still see it
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "name": self.name,
+            "ok": self.ok,
+            "ms": round(self.ms, 1),
+            "detail": self.detail,
+            "error": self.error,
+        }
+
+
+class Turn:
+    """One user utterance and everything the AI did in response."""
+
+    def __init__(self, monitor: "Monitor", turn_id: int, source: str) -> None:
+        self._monitor = monitor
+        self.id = turn_id
+        self.source = source
+        self.wall = _now_wall()
+        self._t0 = _now_mono()
+        self.heard_text = ""
+        self.reply_text = ""
+        self.status = "active"  # active | ok | error
+        self.error = ""
+        self.total_ms = 0.0
+        self._steps: list[Step] = []
+        self._error_logged = False  # count this turn's failure at most once
+
+    # -- recording API (called from the pipeline) ------------------------- #
+    def heard(self, text: str) -> None:
+        self.heard_text = text
+        self._touch()
+
+    def replied(self, text: str) -> None:
+        self.reply_text = text
+        self._touch()
+
+    def step(self, name: str) -> Step:
+        return Step(self, name)
+
+    def finish(self, error: str = "") -> None:
+        self.total_ms = (_now_mono() - self._t0) * 1000.0
+        if error:
+            self.status = "error"
+            self.error = error
+        elif any(s.ok is False for s in self._steps):
+            self.status = "error"
+            if not self.error:
+                failed = next((s for s in self._steps if s.ok is False), None)
+                self.error = (failed.error if failed else "") or "step failed"
+        else:
+            self.status = "ok"
+        # A turn that ended in error must be reflected in errors_total. That
+        # counter is driven by error-level log events on BOTH the server
+        # (Monitor.log) and the browser (dashboard SSE handler), so emit one
+        # log event here rather than bumping a counter the client won't mirror.
+        # Guarded so the repeated _touch()/finish() calls can't double-count.
+        if self.status == "error" and not self._error_logged:
+            self._error_logged = True
+            self._monitor.log("error", f"대화 #{self.id} 실패: {self.error}")
+        self._touch()
+
+    # -- internal --------------------------------------------------------- #
+    def _touch(self) -> None:
+        self._monitor._publish(self)
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "id": self.id,
+            "source": self.source,
+            "wall": self.wall,
+            "heard": self.heard_text,
+            "reply": self.reply_text,
+            "status": self.status,
+            "error": self.error,
+            "total_ms": round(self.total_ms, 1),
+            "steps": [s.to_dict() for s in self._steps],
+        }
+
+
+class Monitor:
+    """Rolling record of turns + status, with a pub/sub for live updates."""
+
+    def __init__(self, keep: int = 60) -> None:
+        self._turns: deque[Turn] = deque(maxlen=keep)
+        self._events: deque[dict[str, Any]] = deque(maxlen=200)
+        self._status: dict[str, Any] = {
+            "running": False,
+            "listening": False,
+            "started_wall": _now_wall(),
+            "components": {},
+            "turns_total": 0,
+            "errors_total": 0,
+        }
+        self._lock = threading.Lock()
+        self._subs: list["queue.Queue[str]"] = []
+        self._id = 0
+
+    # -- status ----------------------------------------------------------- #
+    def set_status(self, **kw: Any) -> None:
+        with self._lock:
+            self._status.update(kw)
+        self._broadcast({"type": "status", "status": self.status_snapshot()})
+
+    def set_components(self, components: dict[str, Any]) -> None:
+        with self._lock:
+            self._status["components"] = components
+        self._broadcast({"type": "status", "status": self.status_snapshot()})
+
+    def status_snapshot(self) -> dict[str, Any]:
+        with self._lock:
+            s = dict(self._status)
+        s["uptime_s"] = round(_now_wall() - s["started_wall"], 1)
+        return s
+
+    def log(self, level: str, message: str) -> None:
+        """A free-form lifecycle/error line (startup, disconnect, crash…)."""
+        evt = {"type": "log", "level": level, "message": message, "wall": _now_wall()}
+        with self._lock:
+            self._events.append(evt)
+            if level == "error":
+                self._status["errors_total"] += 1
+        self._broadcast(evt)
+
+    # -- turns ------------------------------------------------------------ #
+    def turn(self, source: str = "voice") -> Turn:
+        with self._lock:
+            self._id += 1
+            self._status["turns_total"] += 1
+            t = Turn(self, self._id, source)
+            self._turns.append(t)
+        self._publish(t)
+        return t
+
+    def _publish(self, t: Turn) -> None:
+        # errors_total is bumped once when the turn transitions to error, inside
+        # Turn.finish() (via a log event), so this only streams the turn state.
+        self._broadcast({"type": "turn", "turn": t.to_dict()})
+
+    # -- snapshot / subscribe (read side, HTTP threads) ------------------- #
+    def snapshot(self) -> dict[str, Any]:
+        with self._lock:
+            turns = [t.to_dict() for t in self._turns]
+            events = list(self._events)
+        return {
+            "status": self.status_snapshot(),
+            "turns": turns,
+            "events": events,
+        }
+
+    def subscribe(self) -> "queue.Queue[str]":
+        q: "queue.Queue[str]" = queue.Queue(maxsize=256)
+        with self._lock:
+            self._subs.append(q)
+        return q
+
+    def unsubscribe(self, q: "queue.Queue[str]") -> None:
+        with self._lock:
+            if q in self._subs:
+                self._subs.remove(q)
+
+    def _broadcast(self, event: dict[str, Any]) -> None:
+        data = json.dumps(event, ensure_ascii=False)
+        with self._lock:
+            subs = list(self._subs)
+        for q in subs:
+            try:
+                q.put_nowait(data)
+            except queue.Full:
+                # Slow client: drop it rather than block the pipeline.
+                self.unsubscribe(q)

wsai/pipeline.py

@@ -0,0 +1,196 @@
+"""Orchestrator: wires the perception loop and the conversation loop together."""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+
+from .interfaces import (
+    Brain,
+    FrameSource,
+    Reply,
+    SpeechToText,
+    TextChannel,
+    TextToSpeech,
+    Utterance,
+    VisionBackend,
+)
+from .monitor import Monitor
+from .state import SharedScreenContext
+
+log = logging.getLogger("wsai.pipeline")
+
+
+class Pipeline:
+    """Runs two concurrent loops:
+
+    * perception: FrameSource -> VisionBackend -> SharedScreenContext
+    * conversation: (SpeechToText | TextChannel) -> Brain -> (TextToSpeech | TextChannel)
+
+    Any half can be omitted. With no source/vision it runs eyes-free as a pure
+    voice loop (STT -> Brain -> TTS); with no stt/tts it runs text-only; with no
+    conversation it is a headless "just watch" configuration.
+    """
+
+    def __init__(
+        self,
+        *,
+        source: FrameSource | None = None,
+        vision: VisionBackend | None = None,
+        brain: Brain,
+        stt: SpeechToText | None = None,
+        tts: TextToSpeech | None = None,
+        text_channel: TextChannel | None = None,
+        history_turns: int = 12,
+        monitor: Monitor | None = None,
+    ) -> None:
+        self.source = source
+        self.vision = vision
+        self.brain = brain
+        self.stt = stt
+        self.tts = tts
+        self.text_channel = text_channel
+        self.monitor = monitor
+        self.context = SharedScreenContext()
+        self._history: list[tuple[str, str]] = []
+        self._history_turns = history_turns
+
+    # -- perception -------------------------------------------------------- #
+    async def _perceive(self) -> None:
+        if self.source is None or self.vision is None:
+            return  # eyes-free (voice-only) configuration
+        async for frame in self.source.frames():
+            try:
+                obs = await self.vision.describe(frame)
+            except Exception as exc:  # a single bad frame must not kill the loop
+                log.exception("vision.describe failed")
+                if self.monitor is not None:
+                    self.monitor.log("error", f"화면 이해 실패: {exc}")
+                continue
+            await self.context.update(obs)
+            log.debug("screen: %s", obs.text[:120])
+
+    # -- conversation ------------------------------------------------------ #
+    async def _handle(self, utt: Utterance) -> None:
+        if self.monitor is None:
+            screen = await self.context.latest()
+            reply = await self.brain.respond(utt.text, screen, self._history)
+            self._remember(utt.text, reply.text)
+            await self._emit(reply)
+            return
+
+        # Same work, but each stage is timed and streamed to the dashboard so a
+        # viewer can see what was heard, what the brain answered, how long each
+        # step took, and whether anything errored.
+        turn = self.monitor.turn(source=utt.source)
+        turn.heard(utt.text)
+        try:
+            async with turn.step("화면 맥락"):
+                screen = await self.context.latest()
+            async with turn.step("두뇌(생각)"):
+                reply = await self.brain.respond(utt.text, screen, self._history)
+            turn.replied(reply.text)
+            self._remember(utt.text, reply.text)
+            async with turn.step("응답(TTS/전송)"):
+                await self._emit(reply)
+        except Exception as exc:
+            # finish() records the error and emits the single error-level log
+            # event that bumps errors_total, so don't log the same failure twice.
+            turn.finish(error=f"{type(exc).__name__}: {exc}")
+            raise
+        else:
+            turn.finish()
+
+    def _remember(self, user: str, ai: str) -> None:
+        self._history.append((user, ai))
+        if len(self._history) > self._history_turns:
+            self._history = self._history[-self._history_turns :]
+
+    async def _emit(self, reply: Reply) -> None:
+        tasks = []
+        if self.tts is not None:
+            tasks.append(self.tts.speak(reply))
+        if self.text_channel is not None:
+            tasks.append(self.text_channel.send(reply))
+        if not tasks:
+            log.info("AI: %s", reply.text)
+        else:
+            await asyncio.gather(*tasks)
+
+    async def _listen_voice(self) -> None:
+        if self.stt is None:
+            return
+        if self.monitor is not None:
+            self.monitor.set_status(listening=True)
+            self.monitor.log("info", "음성 수신 시작 — 발화 대기 중")
+        try:
+            async for utt in self.stt.utterances():
+                await self._handle(utt)
+        finally:
+            if self.monitor is not None:
+                self.monitor.set_status(listening=False)
+
+    async def _listen_text(self) -> None:
+        if self.text_channel is None:
+            return
+        async for utt in self.text_channel.messages():
+            await self._handle(utt)
+
+    # -- lifecycle --------------------------------------------------------- #
+    async def _prewarm(self) -> None:
+        """Load slow-to-start backends before the loops accept input.
+
+        Real STT/TTS backends (faster-whisper, MeloTTS) load a model into a
+        persistent worker on first use — several seconds on CPU. Warming them
+        here means the first real utterance is answered warm (~1s) instead of
+        paying the cold model load mid-conversation."""
+        warmers = []
+        for comp, name in ((self.tts, "tts"), (self.stt, "stt")):
+            warmup = getattr(comp, "warmup", None)
+            if callable(warmup):
+                warmers.append((name, warmup))
+        if not warmers:
+            return
+        for name, warmup in warmers:
+            try:
+                await warmup()
+            except Exception as exc:  # a warm failure must not abort startup
+                log.warning("prewarm %s failed: %s", name, exc)
+                if self.monitor is not None:
+                    self.monitor.log("error", f"{name} 예열 실패: {exc}")
+
+    async def run(self) -> None:
+        # A TaskGroup (not bare gather) so that if ONE loop raises, the others
+        # are cancelled and awaited before teardown. With plain gather the
+        # failing loop propagated while the siblings kept running detached, and
+        # aclose() in the finally then closed a source/stt out from under a
+        # still-live loop (close-during-use).
+        if self.monitor is not None:
+            self.monitor.set_status(running=True)
+            self.monitor.log("info", "파이프라인 시작")
+        await self._prewarm()
+        try:
+            async with asyncio.TaskGroup() as tg:
+                tg.create_task(self._perceive())
+                tg.create_task(self._listen_voice())
+                tg.create_task(self._listen_text())
+        except* Exception as eg:
+            if self.monitor is not None:
+                for exc in eg.exceptions:
+                    self.monitor.log("error", f"루프 예외: {type(exc).__name__}: {exc}")
+            raise
+        finally:
+            if self.monitor is not None:
+                self.monitor.set_status(running=False, listening=False)
+                self.monitor.log("info", "파이프라인 종료")
+            await self.aclose()
+
+    async def aclose(self) -> None:
+        # tts is included because a real TTS (e.g. MeloTTS) owns a worker
+        # subprocess that must be torn down; mock backends have no aclose.
+        for closer in (self.source, self.stt, self.text_channel, self.tts):
+            if closer is not None and hasattr(closer, "aclose"):
+                try:
+                    await closer.aclose()
+                except Exception:
+                    log.exception("error closing %s", closer)

wsai/state.py

@@ -0,0 +1,34 @@
+"""Shared, thread/async-safe screen context.
+
+The perception loop keeps writing the latest ScreenObservation here; the
+conversation loop reads it when the user says something. We only keep the most
+recent observation plus a short ring buffer of recent ones so the Brain can
+notice "the screen changed" without us re-sending every frame.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections import deque
+
+from .interfaces import ScreenObservation
+
+
+class SharedScreenContext:
+    def __init__(self, history: int = 8) -> None:
+        self._latest: ScreenObservation | None = None
+        self._recent: deque[ScreenObservation] = deque(maxlen=history)
+        self._lock = asyncio.Lock()
+
+    async def update(self, obs: ScreenObservation) -> None:
+        async with self._lock:
+            self._latest = obs
+            self._recent.append(obs)
+
+    async def latest(self) -> ScreenObservation | None:
+        async with self._lock:
+            return self._latest
+
+    async def recent(self) -> list[ScreenObservation]:
+        async with self._lock:
+            return list(self._recent)