The transcript channel only showed STT and LLM seconds. Add wall-clock
start/end times and durations for listening, LLM and TTS so it's obvious
what takes long; STT surfaces as the gap between listening end and LLM start.
- bridge: emit llm_start_ms/llm_end_ms on meta and tts_*_ms on the end event
- bot: capture the listening window, assemble full timing after the stream,
and render a per-stage breakdown in the transcript message
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Proven approach: the conversation (hear+speak) runs on @discordjs/voice; the
Go-Live broadcast is a SEPARATE stream connection created on the SAME selfbot
session (exactly like a real Discord client), so ONE account hears, speaks, AND
broadcasts — no second login, no self_deaf, no voice conflict.
- voice.ts captures its own voice session_id (adapter wrap) and exposes
getSharedSession() {client, guildId, channelId, sessionId, botId}.
- broadcast.ts threads it into the StreamContext.
- selfbot.ts: when a shared session is present, build the Streamer on the
conversation client and create the stream on its session_id (no login/joinVoice/
humanPause); teardown only stops the stream (never leaves voice/destroys the
shared client). Falls back to the dedicated-account path otherwise.
Verified live: Go-Live connected in ~7s while the conversation voice stayed
ready, and the broadcast was visible in Discord — all on one account.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The transcript channel only showed successful transcripts, so dropped utterances
(the 47/50 misses) were invisible. Now every captured utterance is mirrored with
its outcome and per-stage timing:
- too-short blip (<300ms), VAD "음성 아님(VAD 차단)", "인식 실패", "답변 없음", or "ok"
- transcript + reply (or "(무응답)")
- ⏱️ stt/llm seconds
The bridge meta now carries note + stt_sec + think_sec; voice.ts fires onTurn for
every turn (not only non-empty transcripts) and for the too-short drop; userbot
formats the diagnostic line.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Set DISCORD_TRANSCRIPT_CHANNEL_ID to a text channel and the userbot posts
"들은 말: <transcript> / 답변: <reply or (무응답)>" for every voice turn, so you can
verify what the bot understood even when it doesn't answer aloud.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Whisper 'small' frequently mis-transcribes Korean (esp. the wake word 자비스:
'아비스', '자 비싸'). STT is only ~0.1s warm so there is ample latency headroom;
'medium' is far more accurate and fits VRAM alongside qwen2.5:3b on the 8GB GPU.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Warm per-turn timing showed STT 0.1s, TTS ~1-3s, but the reply engine (LLM) was
8-17s — even a simple "고마워" took 16.7s — because it makes multiple model calls
per turn. Add a PLANNER_ENABLED env override (config.py) and default it to 0 in
the userbot compose so the pre-loop planner's extra LLM round-trip is dropped on
this latency-sensitive voice deployment. Also pins STT_LANGUAGE=ko in compose.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- transcribe() now passes language="ko" (STT_LANGUAGE env, default ko): skips
Whisper auto-detect, fixing occasional Korean->Chinese mis-detection and
shaving latency. LLM is already locked via OUTPUT_LANGUAGE=Korean; MeloTTS is
Korean-only — so STT/LLM/TTS are all Korean now.
- converse_stream logs "⏱️ turn stt=.. think(LLM)=.. tts=.. total=.." so the
~30s voice-reply latency can be attributed to the real bottleneck stage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live test showed the bot needed ~30 tries to answer once: 15 of 18 captured
utterances were dropped as "segment dropped (confidence=...)" — but they were
real speech ("자비스 안녕" transcribed at avg_logprob-confidence 0.0-0.29), killed
by the min_confidence=0.3 floor. Short/quiet/accented Korean over a Discord mic
scores very low avg_logprob, so the confidence floor eats real speech.
Noise rejection is handled by the VAD pre-gate + no_speech_prob hard cutoff
(both kept). The avg_logprob floor is now OFF by default (STT_MIN_CONFIDENCE=0,
env-tunable), and the no_speech threshold is env-tunable too. Raise only if
hallucinations slip past the no_speech gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Critical regression: in single-account userbot mode the broadcast auto-start
logged in a SECOND session of the conversation account and called joinVoice,
which the streaming lib always sends with self_deaf:true. Voice state is
per-account, so this deafened the CONVERSATION session too and the bot silently
stopped hearing the user (STT receive broke). The Go-Live cannot connect on a
shared account anyway.
start() now refuses early (no joinVoice, no deafen) when there is no dedicated
DISCORD_STREAM_TOKEN and no normal-bot token, protecting the conversation. A
dedicated stream account re-enables the broadcast.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Confirmed root cause of "broadcast doesn't appear in Discord" in userbot mode:
the conversation and the Go-Live broadcaster were the SAME Discord account on
two sessions. Discord allows one voice presence per account, so the broadcaster's
voice connection never connects (state voiceReady:false). Proven by isolating the
broadcaster: alone it connects ("Go-Live WebRTC connected"); alongside the
conversation it times out.
The broadcaster now logs in with DISCORD_STREAM_TOKEN when set (a second burner
account dedicated to streaming), falling back to DISCORD_SELFBOT_TOKEN (correct
for normal-bot mode). When userbot mode shares one account it warns loudly with
the fix. Documents the var in .env.example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
isActive() was a local flag set at start() time, and start() fired playStream
without awaiting the actual stream connection, so 'broadcasting' was reported
even when nothing reached Discord (auto-start log + ffmpeg/NVENC running are not
proof of transmission).
start() now waits for the streaming library's real readiness signal (the stream
connection's WebRTC reaching "connected") before declaring live. On timeout it
logs a compact connection-state diagnostic, tears the local ffmpeg pipeline down
immediately, and returns an explicit failure. isActive() reports the real live
state. Timeout is config-driven (STREAM_READY_TIMEOUT_MS, default 25s). Adds a
test for the timeout/teardown path and updates the existing leak-teardown test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Makes the Go-Live auto-start verifiable from the container logs (the reviewer
flagged that a successful broadcast left no log record), alongside the existing
loud failure log.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Address review of the noise/broadcast fixes:
- STT now refuses to run Whisper on non-speech. transcribe() runs the Silero
VAD (bundled with faster-whisper, no new dep) BEFORE the model, so noise or a
brief loud blip with no real speech never reaches STT and can't be
hallucinated into a transcript. The no_speech_prob/avg_logprob post-filter
stays as a second line of defence (a clap the VAD lets through is still killed
by Whisper's own no_speech_prob). VAD is env-tunable (VAD_THRESHOLD,
VAD_MIN_SPEECH_MS, VAD_ENABLED) and fail-open so a VAD error never swallows a
real utterance. Validated on real audio: synthesised Korean speech passes;
silence, a 50ms blip and white noise are rejected.
- Broadcast auto-start no longer blocks the voice join and no longer silently
swallows failures: wiring is synchronous, the Go-Live start runs in the
background with a bounded retry and a loud final-failure log.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bridge transcribe path joined every Whisper segment unconditionally, so a
brief loud sound or background noise that momentarily opened the mic gate (no
real speech) still produced a transcript, and Whisper's noise hallucinations
("감사합니다", "MBC 뉴스", ...) made the bot reply to nothing.
Add bridge/stt_filter.py mirroring the desktop listener's _filter_noisy_segments
policy: a hard no_speech_prob cutoff (whisper_no_speech_threshold) plus an
avg_logprob confidence floor (whisper_min_confidence), both config-driven. Apply
it in transcribe() so only segments that look like human speech survive; a
noise-only turn yields an empty transcript and the existing empty-transcript
guard drops it with no reply. Add unit tests for the gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "couple broadcast to voice" feature only wired auto-start into the
normal-bot join handler. Userbot mode (the only mode that can actually Go
Live) was added later with Go-Live deferred to a "stage 2" that never
landed, so the running deployment had no path to start a broadcast.
Extract the coupling into a shared bot/src/broadcast.ts (auto-start on
join, report live state to the brain each turn, voice toggle) and wire it
into both index.ts and userbot.ts so both modes behave identically. Add a
behavioural test for the auto-start + toggle contract.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The streaming splitter treated every "." as a sentence boundary, so the
operational reply "17.5°C" was read as "17." / "5°C" and "1.8 km/h" as
"1." / "8 km/h" - numbers spoken digit-by-digit plus extra TTS calls.
An ASCII terminator (. ! ?) now only ends a sentence when it is followed by
whitespace, a closing quote/bracket, or end of text. In-token dots (decimals
"17.5", versions "v2.0", hosts "example.com") are followed by a digit/letter,
so they no longer split. CJK fullwidth terminators stay unconditional since
those scripts use no trailing space. Language-agnostic, punctuation only.
- bridge: lookahead-gated boundary regex + finditer-based chunking
- tests: regression cases for decimals (17.5/1.8), versions, URLs, and an
integer that genuinely ends a sentence
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The /converse turn synthesised the entire reply before any audio played, so
time-to-first-audio grew with reply length. Add a streaming /converse_stream
endpoint that emits the transcript/reply first, then one audio clip per
sentence as each finishes synthesising. The Discord voice layer enqueues each
clip on arrival via the existing FIFO playQueue, so the first sentence starts
speaking while the rest are still being synthesised.
STT and the reply engine still run to completion before the first clip; only
TTS is pipelined. The non-streaming /converse and /text endpoints are
unchanged.
- bridge: language-agnostic sentence splitter (bridge/text_utils.py) + NDJSON
streaming route
- bot: ndjson() reader + converseStream() client; voice.ts plays clips
progressively
- tests: splitter unit tests + bot ndjson/converseStream tests
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Empirical A/B/C measurement against the live RTX 5050 Ollama stack
(qwen2.5:3b + nomic-embed-text) showed keep_alive=0 unloads the embed
model ~2s after every call, so each turn after a brief idle gap pays a
cold reload. VRAM is not the constraint (~4.4-4.7 GB free with both
models resident) and keep_alive=0 never evicted the chat model, so CPU
embedding (num_gpu=0) gave no benefit. A short positive keep_alive is
the fastest of the three: it keeps the ~0.3 GB embed model resident
across consecutive turns at negligible VRAM cost.
Add tests/test_embeddings.py covering the warm-across-turns behaviour.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "디스코드 토큰은 마지막에" section still told users to fill DISCORD_BOT_TOKEN
and call the `/자비스` slash command, but the default mode is the userbot
(selfbot) — a normal bot account cannot Go Live. Rewrite the setup to lead with
DISCORD_SELFBOT_TOKEN + DISCORD_VOICE_CHANNEL_ID and the text-command control
(`!자비스 join`/`leave`), with the burner-account ToS warning, and keep the
normal-bot path documented as the optional legacy alternative.
The bot, bridge and melo worker boot together, but the MeloTTS model takes
tens of seconds to load. If the bot logged in and auto-joined the voice channel
before the voice was warm, the first reply synthesised to nothing and was
silently dropped.
- bridge /health now reports `tts_ready`. For MeloTTS this pings the worker,
which only binds its HTTP port AFTER the model is loaded (main() warms before
serve_forever()), so a successful ping is a precise "voice is warm" signal.
- The bot polls /health and waits for `tts_ready` before logging in. It does
not wait on brain_ready (the reply engine / Whisper load lazily on the first
turn — a slow first turn is fine, a silent one is the bug). After a 180s cap
it proceeds anyway so a TTS load failure degrades to text-only.
Live-verified: startup logs show "⏳ MeloTTS 준비 대기 중" then
"✓ 음성(MeloTTS) 준비 완료 — 로그인 진행" then "✓ 유저봇 로그인", in that order.
Open-Meteo's geocoder only matches Latin/English spellings, so a Korean city
name like "서울" returns zero results even though the place exists. With
OUTPUT_LANGUAGE locked to Korean the tool-calling model naturally fills
`location` with the Korean name, which dead-ended on every weather request
("could not find location").
When the first geocode is empty and the name is non-ASCII, ask the warm small
model for the common English exonym ("서울" -> "Seoul") and retry once. ASCII
names skip the round-trip entirely.
Live-verified: "서울 날씨 알려줘" now returns real Seoul weather. Tests cover the
romanise-and-retry path and the ASCII short-circuit.
The config template never set intent_judge_model, so it fell back to the code
default gemma4:e2b. That model is not pulled by this stack (Ollama only has
qwen2.5:3b, qwen3:8b, nomic-embed-text), so every auxiliary small-model call —
intent judge, tool router, weather place extraction, query decomposition —
targeted a non-existent model, silently failed, and fell open. This crippled
tool routing and argument extraction on the 3B brain.
Render intent_judge_model from a new OLLAMA_INTENT_MODEL env var that defaults
to OLLAMA_CHAT_MODEL, so the auxiliary calls reuse the already-warm chat model
(one resident model, no extra load). tool_router_model="" then resolves through
the chain to the same model.
Verified: rendered jarvis.json now has intent_judge_model=qwen2.5:3b, and the
weather place extractor returns "서울" / "Tokyo" (it returned None for
everything while pointed at the missing gemma4:e2b).
qwen2.5:3b emits tool calls in text shapes the parser dropped, breaking
two reviewer-reported behaviours:
- `getWeather: {"location": "Seoul"}` (a JSON object after the colon) was
dumped wholesale into {"query": "{...}"}, so `location` never reached the
tool. getWeather then ran with empty args, returned the auto-detected
location's weather, the model noticed the mismatch and retried — looping up
to 8 times before giving up with an English error. Now the JSON object after
the colon is parsed directly as the argument dict.
- `call_stop: {"id":..., "function": {"name": "setBroadcast",
"arguments": "{\"action\": \"stop\"}"}}` — a single tool_call object without
the `tool_calls: [...]` array wrapper, behind a `call_xxx:` label — matched
no form, so the raw JSON leaked to the user AND setBroadcast never ran
("방송 꺼줘" did nothing). Now name + arguments are pulled from the embedded
`function` object when the name is in the allow-list.
Field-captured from the live qwen2.5:3b brain (2026-06-12). Tests cover both
shapes, non-ASCII args, dict/string arguments, and unknown-tool rejection.
Harden the reply-language lock so qwen2.5:3b reliably stays in the locked
language instead of leaking the query language back in:
- reply_language_directive(): single resolver with clear precedence —
explicit OUTPUT_LANGUAGE lock wins over the Piper/Chatterbox English-only
fallback (this deployment's actual TTS is Korean MeloTTS, so the legacy
English lock was both wrong and contradicting the Korean lock).
- Stronger, override-explicit directive wording, inserted near the FRONT of
the system prompt so a small model gives it primacy over the persona.
- build_system_prompt(output_language=...): rewrite the persona's "in the
user's language" clause to the locked language so the persona stops
fighting the lock.
- docs/llm_contexts.md: document the resolver, precedence, and placement.
Live-verified on the running brain (qwen2.5:3b): Korean voice-style input
and a cold English query both return fully Korean replies with no CJK/Hanja
leak. Tests cover unset/set/agnostic/whitespace + precedence + persona rewrite.
Add an optional OUTPUT_LANGUAGE env var that forces every reply into a
single language. When set, output_language_directive() injects a "respond
only in <language>" instruction (also forbidding other scripts) into the
chat loop's system prompt, next to the existing TTS English-only lock.
Empty (default) keeps the multilingual "reply in the user's language"
behaviour, so upstream is unaffected.
For the Korean-only deployment this also suppresses the occasional trailing
CJK/Hanja fragment qwen2.5:3b leaks on free-form chit-chat.
- system_prompt.py: language-agnostic output_language_directive() helper
- engine.py: read OUTPUT_LANGUAGE, append directive in _build_initial_system_message
- docker-compose.yml + .env.example: document/pass the new var
- docs/llm_contexts.md: note the new gating on the main reply context
- tests: cover unset/set/agnostic/whitespace cases
Replace the blunt global OLLAMA_KEEP_ALIVE=-1 (which kept every model,
including nomic-embed, resident in VRAM forever) with per-request residency:
- llm.py: all three /api/chat payloads send keep_alive=30m so the actively
used chat model stays resident and voice turns never pay a cold reload.
- embeddings.py: /api/embeddings sends keep_alive=0 so nomic-embed unloads
right after each call instead of squatting in VRAM next to the chat model.
- docker-compose.yml: drop the global OLLAMA_KEEP_ALIVE=-1; document the
per-request scheme on the ollama service.
Switch the default chat model qwen3:8b -> qwen2.5:3b. Verified live on the
RTX 5050 (8GB):
- ollama ps: qwen2.5:3b 2.4GB, 100% GPU (8B was 92% GPU / 8% CPU), UNTIL ~30m
(the 30m pin, not "Forever"); nomic-embed absent after several enriched turns.
- nvidia-smi: ~3.2GB VRAM used total (qwen 2.4GB + whisper 0.7GB) vs ~6.6GB.
- Korean /text turns: warm 1.7-4s (cold first load ~52s), vs ~5-7s on 8B;
time/weather/places tool calls fire and reply in Korean.
Known limitation: qwen2.5:3b can occasionally leak a trailing CJK phrase on
free-form chit-chat (factual/tool replies stay clean).
Default ollama keep_alive is 5 min, so any voice turn after a short idle paid
a full Qwen3 8B reload into the GPU (~30-60s) before replying. Set
OLLAMA_KEEP_ALIVE=-1 so the chat + embed models stay loaded.
Measured: post-idle turn ~60s -> steady-state ~4.5s per turn (model pinned
"Forever" in `ollama ps`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a dedicated MeloTTS Korean voice (speed 1.5) as the primary TTS engine,
served by a long-lived in-container worker so each Discord turn pays only
inference cost, not model-load cost.
- bridge/melo_worker.py: tiny HTTP service in its own /opt/melo py3.11 venv,
keeps the KR model warm, returns PCM16 WAV on POST /synth.
- bridge/server.py: synthesize() routes to the melo worker first; Piper stays
as an opt-in fallback (MELO_FALLBACK_PIPER, default off so Korean is never
mangled through the English voice). /health reports tts_engine.
- docker/setup-melo.sh: builds the isolated venv (pinned torch 2.12.0 /
torchaudio 2.11.0 CPU, MeloTTS pinned to a commit for reproducible rebuilds),
pre-fetches mecab-ko, and warms a dedicated HF cache (/opt/melo-cache) with a
real KR synth so all BERT + KR checkpoint assets are baked into the image.
- docker/supervisord.conf: runs melo-worker before the bridge with
HF_HOME=/opt/melo-cache (the whisper_cache volume shadows the default HF
cache) plus HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE so it reads the baked cache
and never retries the network on load.
- Dockerfile/.env.example: wire the melo build layer and config knobs.
Verified: offline synth passes with --network none and the prod volume mounted;
prod container recreated, all supervisord services up, bot logged in, and an
end-to-end /tts call returns a 44.1kHz mono PCM16 WAV.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The broadcast subtitle rule treated any ko* track as Korean, so YouTube's
auto-generated (자동 생성) Korean track would switch captions on. Match only
human-authored tracks (kind !== 'asr', vss_id not 'a.*') and pass the full
track object to setOption so YouTube selects the manual track rather than the
same-languageCode ASR one. Captions stay off when only an auto track exists.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The selfbot library's native voice connection cannot complete Discord's current
DAVE/v8 voice handshake and times out (15s). Root cause: DAVE E2EE (mandated
~March 2026). The bundled @discordjs/voice 0.18.0 also lacked DAVE.
Fix: upgrade @discordjs/voice to 0.19.2 + add @snazzah/davey (the DAVE protocol
lib) and drive the selfbot client through @discordjs/voice's joinVoiceChannel /
receiver via the client's voiceAdapterCreator — i.e. reuse the normal-bot
VoiceSession with a user-account channel. Verified: the userbot now reaches a
READY voice connection (host test) and joins the channel in-container with no
timeout. This also fixes the normal-bot voice path under DAVE. Drops the
hand-rolled selfbot-native receiver (audio.ts).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A user account is the only kind Discord lets Go Live, so add a userbot run mode:
when DISCORD_BOT_TOKEN is absent but DISCORD_SELFBOT_TOKEN is set, the app runs
as a userbot instead of the normal slash-command bot.
Stage 1 (conversation): log in with discord.js-selfbot-v13, auto-join
DISCORD_VOICE_CHANNEL_ID on startup (or "!자비스 join" when unset), listen via the
selfbot VoiceReceiver (pcm), transcribe+reply through the bridge, and speak the
reply back with playAudio. Shared PCM/WAV helpers extracted to audio.ts.
run-bot.sh now starts in either userbot or normal mode. Go-Live broadcast +
broadcast-coupled routing land in stage 2 on the same voice connection.
ToS note: selfbots violate Discord ToS; burner account only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Addresses review findings on the dockerized stack:
- Container Chrome search was dead: add --remote-debugging-port + a non-default
--user-data-dir (Chrome 136+ refuses CDP on the default profile), add the
playwright dep (browse-search.mjs connectOverCDP) with browser download
skipped, and connect to 127.0.0.1 not "localhost" (container localhost -> ::1
while Chrome binds IPv4). Verified: browse-search returns real results.
- Broadcast toggle reliability: always offer setBroadcast in screen-share mode
(the embedding/keyword router dropped it for non-English utterances) and make
its description force a tool call. "방송 꺼줘"->stop now 5/5; no false triggers.
- Stop the broadcast on voice leave (no orphaned stream).
- Security: bind VNC/noVNC to loopback by default (VNC_BIND override) and the
bridge to the container loopback (BRIDGE_HOST=127.0.0.1), not published.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the STREAM_BROWSER=true behaviour:
- handleJoin auto-starts the broadcast on voice join and wires the session to
the guild streamer; each turn reports the live state to the brain so search
routes Chrome (live) vs Gemini (off).
- New setBroadcast tool lets the user toggle the broadcast by voice ("방송
켜줘/꺼줘") via the LLM (no hardcoded phrases); it refuses when
STREAM_BROWSER=false. The directive flows brain -> bridge (broadcast_action)
-> bot streamer.start/stop, guarded by isActive() so it's idempotent.
- Per-turn IPC uses a thread-local (reply/turn_state.py) instead of threading
params through the whole engine chain: bridge sets broadcasting in, tool
records the directive out; Tool.execute exposes broadcasting on ToolContext.
Bot typecheck clean; brain covered by tests/test_set_broadcast.py (+ existing
routing tests). Specs + docs updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Makes the all-in-one image actually run the new real-time-search features and
closes review gaps:
- Gemini OAuth path: install Node 22 + @google/gemini-cli (pinned 0.46.0) in the
image; mount a DEDICATED host dir (~/.config/javis/gemini) holding only the
OAuth creds to /root/.gemini (not the whole ~/.gemini). Verified in-container:
`gemini -p ... -o json` returns a grounded answer with no API key.
- Broadcast audio: add PulseAudio + a headless null-sink (run-pulse.sh, new
supervisor program); export XDG_RUNTIME_DIR/PULSE_SERVER so Chrome playback
and the selfbot `ffmpeg -f pulse -i @DEFAULT_MONITOR@` share one daemon.
Verified: default sink virtual_speaker, monitor present, ffmpeg capture OK.
- Bind the brain bridge to 127.0.0.1 only (internal, unauthenticated API).
- VNC host port is overridable; this server pins VNC_PORT=5902 (.env) since the
host already runs Xvnc on 5901.
Verified in-container with CDI GPU passthrough: RTX 5050 visible, NVENC
encoders (h264/hevc/av1) available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
STREAM_BROWSER becomes the broadcast *capability* master flag; the live
screen-share state (new ToolContext.broadcasting, passed per turn by the bot)
decides the backend:
- master off -> broadcast disabled, always Gemini
- master on + live on -> on-screen Chrome (visible on the stream)
- master on + live off -> Gemini
context.broadcasting is None outside the voice path (evals, text entry, older
bot) and falls back to the master flag, so current behaviour is unchanged.
This is the brain-side foundation; bot-side wiring (bridge passes broadcast
state, auto-broadcast on voice join, voice on/off toggle) follows.
Specs + docs/llm_contexts.md updated. Covered by
tests/test_web_search_broadcast_routing.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
yolo auto-approves every tool call, so a real-time search query could in
principle trigger write/shell tools. Default approval mode still auto-runs the
CLI's read-only web search in headless mode but never silently approves
destructive tools. Verified end-to-end: a grounded query returns a current
answer in ~23s with the account OAuth login. Test asserts yolo is absent;
specs and docs/llm_contexts.md updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a GEMINI_AUTH=oauth (default) sub-mode that shells out to the Gemini CLI
using the user's Google-account login instead of an API key. gemini_cli_search()
runs `gemini -p <query> -o json --skip-trust --approval-mode yolo`, strips
GEMINI_API_KEY/GOOGLE_API_KEY and sets GOOGLE_GENAI_USE_GCA=true so the CLI
selects the account OAuth method and fails fast when no login exists. Bounded by
a 30s timeout and fail-open to the DDG/Brave/Wikipedia cascade on any failure
(CLI missing, login expired, quota 429, timeout). GEMINI_AUTH=apikey keeps the
legacy REST path. Specs and docs/llm_contexts.md updated; behaviour covered by
tests/test_realtime_gemini_cli.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the two info modes in the Python brain:
- config.py: read STREAM_BROWSER / GEMINI_API_KEY / GEMINI_MODEL from env into
Settings (stream_browser, gemini_api_key, gemini_model). Verified load_settings
reads both modes.
- realtime_search.py: two fail-open backends returning the same fenced
UNTRUSTED-WEB-EXTRACT envelope: browser_search() shells the Node CDP helper to
drive the on-screen Chrome (visible on the broadcast); gemini_search() calls
the Gemini REST API with google_search grounding.
- web_search.run(): routes by mode before the DDG cascade (true->browser,
false->Gemini), falling through to DDG/Brave/Wikipedia on any miss.
- browse_and_play tool: plays a YouTube video on the shared screen (true mode
only); registered in the tool registry.
- specs + docs/llm_contexts.md updated (new Gemini LLM context); CLAUDE.md spec
registry updated.
Verified live against the running Chrome: true-mode webSearch returned real
Google results for "오늘 서울 날씨", browseAndPlay played the IU 밤편지 MV, and
false-mode degrades gracefully on a bad/absent key. A valid GEMINI_API_KEY is
still needed to confirm the real Gemini grounding output.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First increment of the STREAM_BROWSER real-time-info modes (true = browser,
false = Gemini):
- browse-search.mjs: drives the on-screen Chrome via CDP so the action shows on
the broadcast. `search` returns the top Google results (title/url/snippet);
`youtube` plays the first result. Verified live: real-time Seoul weather
results, and IU 'Good Day' MV playback.
- .env.example: GEMINI_API_KEY / GEMINI_MODEL for the false-mode Gemini account.
- docs/stream_browser_modes.md: architecture + integration map (brain config,
the two mode-gated tools, registry, design decisions) for the remaining wiring.
The Python brain wiring (config.py mode/gemini fields, browseAndSearch +
geminiSearch tools, registry, specs, llm_contexts) lands next - it needs a
running brain and a Gemini key to verify, rather than committing untested edits
into the 39k-line engine.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Addresses review of the STREAM_BROWSER / broadcast-defaults work:
- SelfbotStreamer now spawns broadcast-helper.mjs on stream start and kills it on
stop/self-end (alongside capture + keepalive). The ad-skip, subtitle rule and
fullscreen-toolbar-hide are therefore guaranteed broadcast-wide defaults tied
to the broadcast - not a manual process. Fail-open: if node/Chrome deps are
absent the stream runs without the helper. Verified the helper is a child of
the broadcast holder and armed.
- Enforce STREAM_BROWSER at the streamer (start() returns early when
screenBrowser===false), so EVERY caller including stream-hold.ts is voice-only
when it's off, not just the slash command. stream-hold.ts reads STREAM_BROWSER.
- Fix broadcast-helper fullscreen: resolve the window of the tab actually in
HTML5 fullscreen (via its CDP targetId) instead of the first HTTP tab, so the
right Chrome window is toggled when multiple windows exist.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add STREAM_BROWSER (.env) gating screen-share/browser mode. false => the
/자비스 stream command stays voice + API/MCP only (no Go-Live); true (default)
=> screen share as before. (Browser-driven info retrieval in true mode is a
follow-up build; the bot has no browser-control tools yet.)
- Make the two test-time fixes broadcast-wide defaults via broadcast-helper.mjs:
it now also watches every tab for HTML5 fullscreen and toggles Chrome window
fullscreen so the address bar is hidden for ANY video (xfwm4 won't hide it on
'f' alone), restoring on exit. Subtitles were already enforced per video.
scenario.mjs drops its own fullscreen toggle and relies on the helper.
- Revert the test-settings env vars from .env.example (not wanted).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Addresses review of the ad/subtitle work (the ad-skip.mjs -> broadcast-helper.mjs
rename's other half; the prior commit only recorded the deletion):
- ad mute leak: the ad-skipper muted during an ad but never un-muted, so the
main video stayed silent after the first ad. Save the pre-ad muted/playbackRate
and restore them when the ad ends (verified: muted false -> true -> false).
- captions were only applied once when scenario.mjs ran, not for the whole
broadcast. The persistent helper now applies the rule (OFF by default, Korean
ON if offered) per video and ENFORCES it every tick - one-shot did not hold
because YouTube silently re-enabled captions (verified it stays off across 8s).
- ad-skip + captions merged into broadcast-helper.mjs (one CDP process).
- the 60fps MV test now lives in the repo: scenario.mjs gains MV_QUERY (search +
auto-pick the first >=60fps result) and WATCH_SECONDS, plus the
fullscreen-toolbar-hide fix. The broadcast runs via the committed
stream-hold.ts (audio + keepalive), not an out-of-repo copy.
- document the test env vars (CDP_PORT, HOLD_MS, TEST_*, MV_QUERY, WATCH_SECONDS).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Addresses review of the ad/subtitle work:
- ad mute leak: the ad-skipper muted during an ad but never un-muted, so the
main video stayed silent after the first ad. Save the pre-ad muted/playbackRate
and restore them when the ad ends (verified: muted false -> true -> false).
- captions were only applied once when scenario.mjs ran, not for the whole
broadcast. Move the rule (OFF by default, Korean ON if offered) into the
persistent helper so it runs per video, and ENFORCE it every tick - one-shot
did not hold because YouTube silently re-enabled captions (verified it now
stays off across 8s).
- merge ad-skip.mjs + captions into broadcast-helper.mjs (one CDP process).
- the actual 60fps MV test now lives in the repo: scenario.mjs gains MV_QUERY
(search + auto-pick the first >=60fps result) and WATCH_SECONDS, with the
fullscreen-toolbar-hide fix. The broadcast runs via the committed
stream-hold.ts (audio + keepalive), not an out-of-repo copy.
- document the test env vars (CDP_PORT, HOLD_MS, TEST_*, MV_QUERY, WATCH_SECONDS)
in .env.example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds ad-skip.mjs: connects over CDP and injects a watcher into every tab
(current and future) that clicks "Skip ad" the moment it appears, closes overlay
ads, and fast-forwards unskippable ads (seek-to-end + 16x + mute) so they clear
in ~1s. Self-contained (no extension, no hosts/network changes) and reconnects
across Chrome restarts. Documented in the README.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two broadcast-experience improvements:
- Audio: the Go-Live stream was video-only. Capture the desktop sound (the
default PipeWire/Pulse sink monitor, @DEFAULT_MONITOR@) as a second ffmpeg
input and mux AAC into the mpegts; the library re-encodes it to Opus for
Discord. Controlled by STREAM_AUDIO / STREAM_AUDIO_SOURCE (default on). ffmpeg
inherits XDG_RUNTIME_DIR to reach the pulse socket. Verified: the streamer now
reports "Found audio stream" and the monitor carries Chrome audio (~-11 dB).
- Subtitles: in the browse scenario, default captions OFF, but auto-enable a
Korean track when the video offers one (getOption captions tracklist ->
setOption / unloadModule).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
On the streamed VNC desktop (xfwm4), Chrome did not hide its toolbar when a
video entered HTML5 fullscreen via 'f' - the window was full-screen (outerHeight
1080) but the tab/address bar stayed, leaving only 988px of content, so the
address bar bled into the Go-Live broadcast.
Toggle Chrome-initiated browser fullscreen via CDP (Browser.setWindowBounds
windowState fullscreen) around the 'f' step. That reliably hides the toolbar
(innerHeight 1080 vs 988); the toolbar is restored on exit, so normal browsing
still shows it. Verified live: clean full-screen video, no toolbar.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Go-Live broadcast looked badly choppy: video and scrolling stuttered while
the cursor stayed smooth. Root cause is TigerVNC: it only refreshes its
framebuffer while a VNC client is attached, but the broadcast reads that
framebuffer with x11grab (not as a VNC client). With no viewer attached the
captured screen idled at ~1.5 fps (measured 3/30 distinct frames); the cursor
looked smooth only because x11grab overlays the live cursor on every frame.
- Add a headless RFB keepalive (vnc-keepalive.ts) that stays connected for the
life of the stream and requests incremental framebuffer updates at the stream
framerate. SelfbotStreamer starts it on broadcast start and tears it down on
stop/self-end. Measured 3/30 -> 57/60 distinct frames at 60 fps. Fail-open;
authenticates with VNC_PASSWORD or the ~/.config/tigervnc/passwd file.
- Fix a resource leak: when the Go-Live ended on its own, only the active flag
was cleared, leaving the x11grab->nvenc ffmpeg running forever (pinning a CPU
core while no media was transmitted, with only the gateway TCP left and no UDP
media). The self-end path now tears down capture, keepalive and voice like
stop() does.
- Tests for both paths (self-end teardown; keepalive DES auth, port mapping,
password resolution). Add @types/bun so bun:test typechecks; document the
keepalive and recommended Chrome flags in README and .env.example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bringIntoView returned the last boundingBox() unconditionally after the
scroll loop exhausted, so an element still outside the viewport would be
clicked anyway. Validate the final box against the actual viewport bounds
on both axes (innerWidth/innerHeight) and return null otherwise, so
humanClick fails instead of clicking an off-screen coordinate.
