The bridge only logged Whisper's device on the CPU-fallback path, so a
successful GPU (or silent CPU) load was invisible. Print the CTranslate2-
resolved device on success and on the fallback load, so it is verifiable that
STT is actually running on cuda alongside ollama (GPU) and MeloTTS (cuda).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Chrome: --disable-blink-features=AutomationControlled (+ ko-KR) so Google
shows results, not the /sorry/ automation block.
- Settings persist to /data/jarvis-settings.json (survives recreate; entrypoint
re-merges it) AND the runtime config; apply restarts via a DETACHED process so
the HTTP response isn't dropped when the bridge restarts.
- Bridge reads tts_engine from the settings config so the TTS-engine choice
actually applies.
Adds /settings (served by the bridge) to change the LLM model (from installed
Ollama models), Whisper model, TTS engine + MeloTTS speed, output language,
agentic max-turns, thinking mode, and free-form LLM instructions — live, with a
'apply' that restarts the bridge + TTS worker. Settings persist to the runtime
config JSON; engine reads output_language + llm_instructions and the TTS worker
reads melo_speed from it. Bridge port publishable for access.
Log the captured speech-clip duration (녹음/음성) separately from the Whisper
transcription time (STT처리) so it's clear whether a slow turn is the
listening/recording or the transcription, per the user's request.
The startup warm-up loaded qwen at the default context, but the reply
engine chats at num_ctx=8192 — a different Ollama instance — so the first
real turn still cold-reloaded. Warm at OLLAMA_NUM_CTX via /api/chat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The first spoken turn paid a ~10s cold start because Whisper (default
"medium") and the Ollama chat model loaded lazily on the first request.
Warm them (and ping the TTS worker) in a background thread at startup so
the server accepts requests immediately while models load, and the first
real utterance is fast.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The transcript channel only showed STT and LLM seconds. Add wall-clock
start/end times and durations for listening, LLM and TTS so it's obvious
what takes long; STT surfaces as the gap between listening end and LLM start.
- bridge: emit llm_start_ms/llm_end_ms on meta and tts_*_ms on the end event
- bot: capture the listening window, assemble full timing after the stream,
and render a per-stage breakdown in the transcript message
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The transcript channel only showed successful transcripts, so dropped utterances
(the 47/50 misses) were invisible. Now every captured utterance is mirrored with
its outcome and per-stage timing:
- too-short blip (<300ms), VAD "음성 아님(VAD 차단)", "인식 실패", "답변 없음", or "ok"
- transcript + reply (or "(무응답)")
- ⏱️ stt/llm seconds
The bridge meta now carries note + stt_sec + think_sec; voice.ts fires onTurn for
every turn (not only non-empty transcripts) and for the too-short drop; userbot
formats the diagnostic line.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- transcribe() now passes language="ko" (STT_LANGUAGE env, default ko): skips
Whisper auto-detect, fixing occasional Korean->Chinese mis-detection and
shaving latency. LLM is already locked via OUTPUT_LANGUAGE=Korean; MeloTTS is
Korean-only — so STT/LLM/TTS are all Korean now.
- converse_stream logs "⏱️ turn stt=.. think(LLM)=.. tts=.. total=.." so the
~30s voice-reply latency can be attributed to the real bottleneck stage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live test showed the bot needed ~30 tries to answer once: 15 of 18 captured
utterances were dropped as "segment dropped (confidence=...)" — but they were
real speech ("자비스 안녕" transcribed at avg_logprob-confidence 0.0-0.29), killed
by the min_confidence=0.3 floor. Short/quiet/accented Korean over a Discord mic
scores very low avg_logprob, so the confidence floor eats real speech.
Noise rejection is handled by the VAD pre-gate + no_speech_prob hard cutoff
(both kept). The avg_logprob floor is now OFF by default (STT_MIN_CONFIDENCE=0,
env-tunable), and the no_speech threshold is env-tunable too. Raise only if
hallucinations slip past the no_speech gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Address review of the noise/broadcast fixes:
- STT now refuses to run Whisper on non-speech. transcribe() runs the Silero
VAD (bundled with faster-whisper, no new dep) BEFORE the model, so noise or a
brief loud blip with no real speech never reaches STT and can't be
hallucinated into a transcript. The no_speech_prob/avg_logprob post-filter
stays as a second line of defence (a clap the VAD lets through is still killed
by Whisper's own no_speech_prob). VAD is env-tunable (VAD_THRESHOLD,
VAD_MIN_SPEECH_MS, VAD_ENABLED) and fail-open so a VAD error never swallows a
real utterance. Validated on real audio: synthesised Korean speech passes;
silence, a 50ms blip and white noise are rejected.
- Broadcast auto-start no longer blocks the voice join and no longer silently
swallows failures: wiring is synchronous, the Go-Live start runs in the
background with a bounded retry and a loud final-failure log.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bridge transcribe path joined every Whisper segment unconditionally, so a
brief loud sound or background noise that momentarily opened the mic gate (no
real speech) still produced a transcript, and Whisper's noise hallucinations
("감사합니다", "MBC 뉴스", ...) made the bot reply to nothing.
Add bridge/stt_filter.py mirroring the desktop listener's _filter_noisy_segments
policy: a hard no_speech_prob cutoff (whisper_no_speech_threshold) plus an
avg_logprob confidence floor (whisper_min_confidence), both config-driven. Apply
it in transcribe() so only segments that look like human speech survive; a
noise-only turn yields an empty transcript and the existing empty-transcript
guard drops it with no reply. Add unit tests for the gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The /converse turn synthesised the entire reply before any audio played, so
time-to-first-audio grew with reply length. Add a streaming /converse_stream
endpoint that emits the transcript/reply first, then one audio clip per
sentence as each finishes synthesising. The Discord voice layer enqueues each
clip on arrival via the existing FIFO playQueue, so the first sentence starts
speaking while the rest are still being synthesised.
STT and the reply engine still run to completion before the first clip; only
TTS is pipelined. The non-streaming /converse and /text endpoints are
unchanged.
- bridge: language-agnostic sentence splitter (bridge/text_utils.py) + NDJSON
streaming route
- bot: ndjson() reader + converseStream() client; voice.ts plays clips
progressively
- tests: splitter unit tests + bot ndjson/converseStream tests
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bot, bridge and melo worker boot together, but the MeloTTS model takes
tens of seconds to load. If the bot logged in and auto-joined the voice channel
before the voice was warm, the first reply synthesised to nothing and was
silently dropped.
- bridge /health now reports `tts_ready`. For MeloTTS this pings the worker,
which only binds its HTTP port AFTER the model is loaded (main() warms before
serve_forever()), so a successful ping is a precise "voice is warm" signal.
- The bot polls /health and waits for `tts_ready` before logging in. It does
not wait on brain_ready (the reply engine / Whisper load lazily on the first
turn — a slow first turn is fine, a silent one is the bug). After a 180s cap
it proceeds anyway so a TTS load failure degrades to text-only.
Live-verified: startup logs show "⏳ MeloTTS 준비 대기 중" then
"✓ 음성(MeloTTS) 준비 완료 — 로그인 진행" then "✓ 유저봇 로그인", in that order.
Adds a dedicated MeloTTS Korean voice (speed 1.5) as the primary TTS engine,
served by a long-lived in-container worker so each Discord turn pays only
inference cost, not model-load cost.
- bridge/melo_worker.py: tiny HTTP service in its own /opt/melo py3.11 venv,
keeps the KR model warm, returns PCM16 WAV on POST /synth.
- bridge/server.py: synthesize() routes to the melo worker first; Piper stays
as an opt-in fallback (MELO_FALLBACK_PIPER, default off so Korean is never
mangled through the English voice). /health reports tts_engine.
- docker/setup-melo.sh: builds the isolated venv (pinned torch 2.12.0 /
torchaudio 2.11.0 CPU, MeloTTS pinned to a commit for reproducible rebuilds),
pre-fetches mecab-ko, and warms a dedicated HF cache (/opt/melo-cache) with a
real KR synth so all BERT + KR checkpoint assets are baked into the image.
- docker/supervisord.conf: runs melo-worker before the bridge with
HF_HOME=/opt/melo-cache (the whisper_cache volume shadows the default HF
cache) plus HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE so it reads the baked cache
and never retries the network on load.
- Dockerfile/.env.example: wire the melo build layer and config knobs.
Verified: offline synth passes with --network none and the prod volume mounted;
prod container recreated, all supervisord services up, bot logged in, and an
end-to-end /tts call returns a 44.1kHz mono PCM16 WAV.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the STREAM_BROWSER=true behaviour:
- handleJoin auto-starts the broadcast on voice join and wires the session to
the guild streamer; each turn reports the live state to the brain so search
routes Chrome (live) vs Gemini (off).
- New setBroadcast tool lets the user toggle the broadcast by voice ("방송
켜줘/꺼줘") via the LLM (no hardcoded phrases); it refuses when
STREAM_BROWSER=false. The directive flows brain -> bridge (broadcast_action)
-> bot streamer.start/stop, guarded by isActive() so it's idempotent.
- Per-turn IPC uses a thread-local (reply/turn_state.py) instead of threading
params through the whole engine chain: bridge sets broadcasting in, tool
records the directive out; Tool.execute exposes broadcasting on ToolContext.
Bot typecheck clean; brain covered by tests/test_set_broadcast.py (+ existing
routing tests). Specs + docs updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- voice.ts: reply playback is now a FIFO queue (AudioPlayerStatus.Idle drains
it) so concurrent speakers no longer cut each other's replies off.
- selfbot.ts: rewritten against the REAL @dank074/discord-video-stream v6 API
(verified from its d.ts): prepareStream(input, opts, signal)->{command,output},
playStream(output, streamer, {type:"go-live"}, signal), Streamer.joinVoice.
x11grab via customInputOptions; optional NVENC encode (RTX 5050) via exported
`nvenc`. package.json pinned to ^6.0.0 (was a wrong ^4.2.1).
- Dockerfile: dropped the hardcoded python3.12 LD_LIBRARY_PATH. faster-whisper
>=1.1 self-locates the pip CUDA libs; ldconfig (full path, glob) registers
them as a robust fallback. Verified: ld.so cache lists libcublas/libcudnn and
GPU whisper works with LD_LIBRARY_PATH empty.
- bridge: STT resample 48k->16k upgraded from nearest-neighbor to linear
(np.interp).
Verified: tsc clean, image builds, GPU whisper OK via ldconfig, compose valid.
Code review of the bridge/bot/docker work found:
- TTS bug: bridge called PiperVoice.synthesize(text, wav) but that method
returns AudioChunks and takes a SynthesisConfig as its 2nd arg, not a wav
file -> TTS would fail. Switched to synthesize_wav(text, wav_file).
Verified: produces a valid 22050Hz mono WAV.
- run-bot.sh now waits if ANY of DISCORD_BOT_TOKEN/APP_ID/GUILD_ID is missing
(config.ts throws on a missing one), preventing a supervisor crash-loop.
Verified clean: discord.js Events.ClientReady == 'clientReady' (existing
handler correct); image rebuilds.
GPU acceleration is now on by default and verified end-to-end on the
Blackwell RTX 5050 (sm_120):
- Ollama offloads 100% to GPU (log: library=CUDA compute=12.0,
BLACKWELL_NATIVE_FP4=1). compose passes GPU via CDI
(devices: nvidia.com/gpu=all) to both ollama and javis.
- Whisper STT on GPU: faster-whisper>=1.1.0 + nvidia-cublas/cudnn cu12,
LD_LIBRARY_PATH baked into the image. Verified float16 transcribe on
sm_120; bridge auto-falls back to CPU when no GPU is present.
- Model: default chat model -> qwen3:8b (best 8GB-VRAM tool-calling,
~5GB Q4). Embed stays nomic-embed-text.
- README documents the host one-time setup (nvidia-container-toolkit +
`nvidia-ctk cdi generate`) and GPU on/off.
Verified: image builds; GPU visible in both containers via compose;
ollama ps = 100% GPU; faster-whisper cuda OK + CPU fallback OK;
bridge /health 200.
