User requested the voice assistant speak politely. Update the persona to
require 존댓말 on every reply and rewrite the two 반말 output examples
(emotion demo + clarify-question) accordingly. Verified live: replies come
back polite and still one short sentence with emotion tags.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sonnet 5 reaches first token ~0.4s sooner than Sonnet 4.5 but answers the
same voice prompt more verbosely (measured 46-54 vs ~28 output tokens),
which erased the win in total turn time. Add a cached, persona-independent
brevity system block that pulls output back to ~30 tokens, so the faster
first token becomes a faster, lower-variance whole reply.
Measured (16-round interleaved A/B, production-shaped call):
sonnet-4-5 TTFT 1.26s total 2.02s (tail 3.26s) out 29
sonnet-5+brev TTFT 0.85s total 1.70s (tail 2.28s) out 30
- ClaudeBrain: default model claude-sonnet-5 + BREVITY block (cached with
the persona prefix so a dashboard persona edit can't drop it).
- Defaults aligned: config.Settings.anthropic_model and the voice-server
WSAI_BRAIN_MODEL default -> claude-sonnet-5.
- Dashboard: add claude-sonnet-5 to LLM_OPTIONS + JS label; fix stale
restart hint.
- tests/latency_ab.py: reproducible model-latency A/B harness (reads
CLAUDE_CREDENTIALS_PATH; makes live API calls, so not a pytest test).
Streaming TTS was intentionally not added: this bot answers in one sentence,
where sentence-level streaming has no overlap to exploit, and it would
require rearchitecting both the Python endpoint and the node playback.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
STT (whisper_worker): add VAD tuning (speech_pad_ms so soft first/last words
aren't clipped), condition_on_previous_text=False + temperature fallback +
no_speech/logprob/compression thresholds to reject noisy/quiet decodes, drop
per-segment non-speech, and a hallucination guard that blanks Whisper's classic
Korean silence/noise boilerplate ("감사합니다" 등) when no_speech_prob is high.
Barge-in (bot.mjs): stop the bot's TTS only when the speaking (green ring) stays
on for >= WSAI_BARGE_IN_MS (default 700ms), not on the first blip — cancelled if
speaking stops in time. AfterSilence 800->1000ms so trailing soft words finish.
Fragment gate (dashboard): a lone connective filler ("그러면") carries no
answerable intent -> discard as [대기] instead of replying; hidden by the noise
filter like [잡음].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Voice replies were too long and slow (~1.3s LLM). Cap output at 150 tokens
(WSAI_BRAIN_MAX_TOKENS), rewrite the persona to demand one short sentence with
no boilerplate self-intro / "무엇을 도와드릴까요" padding, and mark the static
system prompt with cache_control so repeat turns skip re-processing it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bump the base TTS speed default (WSAI_TTS_SPEED, slider default/label, dashboard
fallbacks) so the bot speaks a bit faster by default.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Default STT model changed small -> medium (WhisperSTT default + service env;
medium pre-cached). UI labels/hints updated.
- Model 적용 buttons: if the picked model == current, flash "변경사항 없음" for 3s
then show the current model again; if it actually changes, STT shows a live
loading panel that polls /api/models until the worker is ready and logs
completion to the event log (set_stt_model/set_llm_model now return `changed`).
- Event/error log gains categories (READY, CONNECT, MODEL, TTS, VOICE, FILTER,
SETTING, TURN, BRAIN, VISION, PIPELINE, …): Monitor.log takes an optional `cat`,
call sites tagged, and each line shows a coloured category chip. A bot
(re)connection now logs a CONNECT event.
- Log search extended: filter by 레벨(type), 종류(category, auto-populated), and
시간(time range) in addition to free text.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Each emotion can now be tuned independently. parse_segments resolves the four
controls per segment from a base (공통) dict plus an optional per-emotion
override; a missing override key inherits base. By default there are no
overrides, so every emotion delivers with the base values (모든 감정 = 기본값).
- emotion.py: Segment now carries all 4 controls + the canonical emotion name;
parse_segments(text, base, overrides). Adds EMOTION_LABELS/EMOTIONS for the UI.
- melo.py: MeloTTS.emotion_overrides store; synth resolves per-segment controls
and sends them per segment.
- melo_worker.py: _render applies each segment's own word_gap/sentence_gap/pitch
(previously reply-global).
- dashboard.py: emotion dropdown in the TTS panel; GET returns base + overrides
+ emotion list; POST {emotion,...} stores an override (or {reset:true} clears
it); base is set when emotion is omitted/"base".
Verified: an override on one emotion slows only that emotion (happy@0.7=5.66s vs
base 3.02s; sad unchanged at 3.06s); dashboard store/reset/base all work; 34
tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Change default voice controls to the user's tuned values: glyph speed 1.35,
word_gap -0.07s (-70ms), sentence_gap -0.30s, pitch 0 (env defaults, slider
initial values, and the 기본값 reset button all updated).
- Make the "봇 목소리(TTS) 조절" dashboard panel collapsible via its header;
starts collapsed (▸), expands on click (▾).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a "봇 목소리(TTS) 조절" panel to the voice-server dashboard so the four
controls can be tuned from the browser instead of only via env vars:
- 4 sliders (glyph speed, word gap, sentence gap, pitch) with live labels
- 미리듣기: synthesises a sample with the slider values WITHOUT changing the
live bot voice (new per-call overrides on MeloTTS.synth)
- 봇에 적용: commits the slider values to the live TTS instance; next reply uses
them. 기본값 button resets to the manual defaults.
Backend: GET/POST /api/tts/settings (clamped to the manual ranges) and POST
/api/tts/preview (returns audio/wav). Panel shows only when tts is a real
(non-mock) backend.
Verified end-to-end against a live dashboard instance: page renders the panel,
GET returns defaults, POST applies+clamps (pitch 99->12), preview returns a
valid wav and leaves the live settings unchanged; 29 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Ports the 4 independent voice controls documented in tts_site's manual into the
MeloTTS backend, layered on top of the existing emotion-tag segments:
- glyph speed: generation-stage length_scale (existing speed path); default
bumped 1.2 -> 1.25 to match the manual (still well below the 1.5 that slurred)
- word_gap (sec, -0.2..0.5): scale intra-sentence silences after natural synth
- sentence_gap (sec, -0.5..1.5): insert/trim silence at sentence boundaries,
replacing the old fixed 120ms inter-segment gap
- pitch (semitones, -12..12): global offset added on top of per-emotion pitch
Each reply is split into sentences, synthesised per-sentence at its segment's
speed, word_gap applied, joined with sentence_gap, then pitch-shifted. Exposed
via WSAI_TTS_SPEED/WORD_GAP/SENTENCE_GAP/PITCH env vars and MeloTTS ctor args.
Verified by real synthesis: each control independently changes wav duration
(speed, sentence_gap, word_gap-with-pauses, pitch); 29 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Voice TTS (MeloTTS) is Korean-only, so any non-Korean reply gets
mangled. Remove the persona's language-switch escape hatch so the AI
always answers in Korean even when the user requests another language.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
User reported the 1.5x base made Korean pronunciation mushy/slurred. Dial the
default WSAI_TTS_SPEED back to 1.2: still noticeably faster than the original
1.0, but clear. Emotion multipliers scale off base as before.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
User reported the emotion samples sounded the same and the speed change wasn't
noticeable. Two changes:
- Add a "base" (기본/neutral) emotion at speed 1.0x so the plain base voice can
be selected explicitly via a [기본] tag and auditioned against the others.
- Bump the WSAI_TTS_SPEED default 1.15 -> 1.5 for a clearly faster base voice.
Emotion multipliers scale off base, so every emotion speeds up together.
Also extends gen_emotion_samples.py to emit one wav per emotion (incl. 기본)
plus a stitched all-in-one, so each emotion can be delivered as a separate clip.
Verified: 29 tests pass; match_emotion('기본') == 'base'; per-emotion synthesis
at base 1.5 produces distinct clip durations (base 4.9s vs happy 3.3s).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
User asked for a bit faster speech. Pitch shift is already disabled, so the
earlier "monster voice" risk from stacking pitch on a fast base is gone — a
modest 1.15x base is natural. Emotion multipliers scale off base, so every
emotion gets the bump too. Also adds tests/gen_emotion_samples.py, a dev
utility that synthesizes one clip per canonical emotion (announce name at
neutral speed, then a sample sentence steered by that emotion's tag) and
stitches them into a single wav for auditioning the emotion palette.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The emotional TTS path applied a librosa post-hoc pitch_shift (±1–3 semitones)
on top of a 1.3x-fast Melo base, producing a robotic "monster" delivery. Zero
out the pitch column for every emotion so pitch_shift is never invoked (the
worker's _pitch_shift already no-ops on 0.0), and drop the default synthesis
speed to 1.0. Emotion is now conveyed by speed alone — natural, artefact-free.
The pitch column is retained so a proper pitch method can be re-enabled later.
Verified: 29 tests pass; real 2-emotion synthesis on CUDA yields a clean wav
with pitch=0.0 on all segments.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The STT/TTS worker _ensure() treated a spawned-but-not-yet-handshaked
subprocess as ready, so a voice turn arriving during warmup read the same
stdout StreamReader concurrently with the warmup handshake and crashed with
"readuntil() called while another coroutine is already waiting for incoming
data". Add a _start_lock + _ready flag so (re)start and the ready handshake
run atomically and callers wait for real readiness before reading stdout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ClaudeBrain now returns per-reply token usage (Reply.usage from the API
response), the dashboard accumulates it (monitor.add_claude_usage), and the
header shows a "클로드 토큰" stat (input+output total, with a tooltip breaking
down input/output tokens and request count).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Redefines the bot as "디스코드를 이용해 실시간으로 대화하는 AI 인공지능" and drops
all screen-share wording. Organizes the one-line prompt into sections (역할·언어·
답변방식·대화태도·안전/사실성·감정표현·정체성). Adds: always-Korean-unless-asked,
길이 정량화(한두 문장·10초), 되묻기/침묵 무시, 불확실·최신 정보는 "확인 필요",
URL·숫자·코드 풀어 읽기. Keeps the [감정] tag section (required by the emotion TTS).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a persisted, runtime-editable system prompt. The brain reads the persona
on every turn (prompt_store.get_persona), so a dashboard edit applies to the
next reply with no restart; blank clears the override back to the built-in
PERSONA. New endpoints GET/POST /api/prompt, and a reusable modal popup
(뒤로가기 + 수정/저장) that later white/blacklist features will share.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Emotion tags now steer delivery rather than being read aloud. parse_segments()
splits a reply on [감정] tags: a recognised emotion word switches the pitch and
speed of the text that follows (and is dropped), while a non-emotion bracket
(e.g. [1번]) keeps its inner words as spoken content. Emotions can change
mid-reply, so a single turn is synthesised as several pitch-shifted segments and
concatenated in the melo worker (librosa pitch_shift, warmed at startup).
The emotion vocabulary is grounded in Azure Neural TTS speaking styles plus
Ekman's basic emotions, with Korean synonyms. The brain persona is updated to
emit inline tags from that set.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Brain persona now prefixes every reply with one bracketed emotion tag
(e.g. [반가움], [궁금]) and keeps replies to one or two short sentences.
- Empty/unrecognised audio (silence/noise) is reported as reply "[잡음]" with
no TTS playback instead of an empty reply.
- TTS speaks the bracketed emotion too: "[힘차게] 안녕!" is synthesised as
"힘차게, 안녕!" via a leading-tag -> spoken-word transform.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude replies with markdown/backticks by default; MeloTTS's Korean text
normaliser has no entry for '`' and dies with KeyError: '`', so any reply
mentioning a command/code block crashed the whole voice turn (500 on
/api/voice-turn). Fix at the shared synth() choke point with
normalize_for_speech(), which flattens code fences/inline code/links/markdown
and guarantees no backtick reaches the worker — covering both the dashboard
voice turn and the Discord speak() bridge. Also add a PERSONA line asking the
model to avoid markdown (belt-and-suspenders; the code strip is the real fix).
errors_total never moved for turn-level failures: it was only bumped by
log("error") events, and the dashboard voice path calls turn.finish(error=...)
without logging. Emit one error-level log event from Turn.finish() when a turn
ends in error, so both the server counter and the browser SSE mirror stay
consistent, guarded to count at most once. Drop the now-redundant pipeline
log("error") to avoid double counting and remove the dead _publish stub.
Verified: raw backtick -> worker KeyError '`' reproduced; after fix real
MeloTTS synth of a backtick+fenced reply succeeds; /api/voice-turn returns 200
with a wav body on a backtick reply and errors_total stays 0, and an induced
synth failure returns 500 with errors_total incrementing to exactly 1. Full
suite 18 passed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A single Claude 529 Overloaded dropped the voice turn straight to the apology
fallback. The anthropic SDK retries >=500/429 but only twice by default, which
a busy window can outlast. Raise max_retries (WSAI_BRAIN_MAX_RETRIES, default 4)
so transient overloads recover silently, and give overloads their own spoken
fallback ("서버가 붐벼서...") distinct from generic failures. Kept modest so a
sustained outage still fails fast instead of leaving the bot silent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bot (dave/bot.mjs) previously only joined the channel and counted audio
frames — it never fed STT or spoke back. Wire the real loop:
- Node bot: buffer each speaker's Opus->PCM utterance until AfterSilence,
wrap as WAV, POST to the Python voice-turn endpoint, then play the returned
reply wav into the channel via an AudioPlayer (ffmpeg->Opus). Skips its own
audio, dedupes overlapping subscriptions, and ignores sub-0.35s noise.
- Python: new `python -m wsai --voice-server` serves /api/voice-turn — decode
the uploaded utterance, GPU faster-whisper STT, produce a reply (echo of what
was heard for now), GPU MeloTTS synth, return the reply wav (recognised/reply
text ride along as X-Heard/X-Reply headers). Both engines pre-warmed; turns
show in the dashboard feed. MeloTTS.synth() extracted for direct wav reuse.
Echo mode verifies listening+speaking+GPU recognition entirely in Discord; the
Claude brain is the next slice. Verified the endpoint round-trip: utterance wav
-> correct Korean X-Heard/X-Reply + a WAVE reply on device=cuda. 12 tests pass,
node --check clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The status page was view-only, so there was no way to actually verify Korean
recognition end-to-end. Add a live test: record from the mic (localhost/https)
or upload an audio file (works over LAN http, where browsers block getUserMedia),
POST it to a new /api/stt endpoint that ffmpeg-normalises the blob to 16 kHz
mono and runs the real GPU faster-whisper, then shows the recognised text +
latency + device. Results also land in the live turn feed.
The dashboard now optionally holds a WhisperSTT and drives it from a private
asyncio loop thread. New `python -m wsai --stt-test` serves the page with STT
enabled and pre-warms the GPU worker so the first recognition is instant.
WhisperSTT.resolved_device is exposed for the UI.
Verified: wav and browser-style webm/opus uploads both return the correct
Korean text on device=cuda in ~240-280ms. 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The first CUDA inference pays a large lazy cost (kernel autotune/cudnn) —
~10s for a cold TTS synth — which would blow the voice loop's ~1s budget on
the very first reply. Each worker now runs one dummy inference (TTS: a short
phrase; STT: 1s of silence) after model load and before emitting "ready", so
"ready" means "hot". Warmup failures are logged and never block startup.
Verified: first real call after startup is now TTS ~238ms / STT ~189ms
(was ~11s cold for TTS). 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both voice backends defaulted to CPU. Fix the "CUDA unavailable" gaps so
everything that benefits from the RTX 5050 uses it:
- MeloTTS venv had CPU-only torch (2.12.0+cpu) -> installed Blackwell-capable
torch/torchaudio 2.11.0+cu128 (sm_120 verified with a real GPU matmul).
- faster-whisper CUDA loaded but transcribe() died with "libcublas.so.12 not
found": installed nvidia-cublas-cu12 + nvidia-cudnn-cu12 into the whisper
venv and inject those nvidia/*/lib dirs into the worker's LD_LIBRARY_PATH at
spawn (the loader only honours it at exec).
- WSAI_WHISPER_DEVICE / WSAI_MELO_DEVICE now default to "auto": pick CUDA when
present, else CPU, and each worker falls back to CPU if a CUDA load fails so
the voice loop never dies on a GPU-less host.
Verified end-to-end through the real backend classes: both workers report
"ready on cuda"; steady-state STT ~170ms (was ~1350ms CPU), TTS ~4s first call
vs ~23s CPU. All 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Step 3 (귀): add WhisperSTT + whisper_worker, a warm out-of-venv worker
mirroring the MeloTTS shape (whisper312 venv, small/int8 on CPU). transcribe()
closes the voice round trip (MeloTTS wav -> whisper text); utterances() turns an
injected audio_source into Utterances (Discord voice feed pending). Wired into
factory as WSAI_STT=whisper.
Also address the arbiter's TTS follow-ups:
- melo worker error handling: capture stderr (drained in a bounded background
task so the pipe can't fill), surface the real failure cause, and defend
against an empty/invalid ready line instead of dying on JSONDecodeError.
- pipeline pre-warm: load slow backends (warmup()) at startup so the first
utterance is answered warm; a warmup failure is logged, not fatal.
Verified: real TTS->STT round trip recovers the sentence near-perfectly;
warm transcribe ~1.2s (CPU). 12 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a MeloTTS backend that runs the model in its own melo311 interpreter
as a long-lived worker (melo_worker.py), loaded once and fed synthesis
requests over a stdin/stdout JSON protocol. fd1 is split from fd2 in the
worker so MeloTTS's stdout progress chatter can't corrupt the protocol.
Each speak() writes a wav and hands the path to a pluggable sink (the
Discord voice step will swap in "play into the call"). factory wires
tts=melo; pipeline.aclose now also tears down the tts worker.
Verified (CPU): model load ~7.9s once, then a short reply synthesizes in
~0.86s (within the ~1s budget); wav is valid 44.1kHz PCM. GPU (cuda) is
selectable via WSAI_MELO_DEVICE for lower latency, pending GPU approval.
7 smoke tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wires the deployment's Claude Max login (OAuth token from
$CLAUDE_CREDENTIALS_PATH) into ClaudeBrain/ClaudeVision. OAuth tokens
authenticate as Bearer (auth_token=), not x-api-key, and only answer
when the first system block is the Claude Code identity string, so the
real persona moves to a second system block. Token is re-read per
request so a host-side refresh is picked up without a restart. Falls
back to ANTHROPIC_API_KEY when set.
Verified: WSAI_BRAIN=claude returns a real Korean reply through the
factory; 7 smoke tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a stdlib-only observability site so you can open a browser and watch,
step by step: whether it is listening, what it heard, what the brain thought
and answered, how long each stage took, and whether anything errored.
- wsai/monitor.py: thread-safe telemetry hub (per-turn timed steps, status
header, error log) with a pub/sub for live push.
- wsai/dashboard.py: stdlib http.server serving a self-contained page plus an
SSE (/events) live stream; /api/state snapshot fallback.
- Pipeline emits step-by-step turn telemetry (화면 맥락 → 두뇌 → 응답) and
listening/running status; optional monitor, so existing paths are untouched.
- `python -m wsai --dashboard` starts the site (0.0.0.0:8787, WSAI_DASHBOARD_PORT)
and loops the mock voice demo so there is always live activity to watch.
- Tests cover turn recording, per-step timing, error marking, and live push.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Modular async pipeline: FrameSource->Vision->context and STT/text->Brain->TTS.
All stages are Protocols; mock backends run end-to-end with no deps/keys.
Real backends included: mss screen capture, Claude vision+brain (guarded imports).
