tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity
Files
11a72cb296b6db81477cddc89ae57d45fe6ac0d6
javis_bot/docs/stream_browser_modes.md
javis-bot ca86390407 feat: couple broadcast to voice + voice-controlled broadcast toggle 
Completes the STREAM_BROWSER=true behaviour:
- handleJoin auto-starts the broadcast on voice join and wires the session to
  the guild streamer; each turn reports the live state to the brain so search
  routes Chrome (live) vs Gemini (off).
- New setBroadcast tool lets the user toggle the broadcast by voice ("방송
  켜줘/꺼줘") via the LLM (no hardcoded phrases); it refuses when
  STREAM_BROWSER=false. The directive flows brain -> bridge (broadcast_action)
  -> bot streamer.start/stop, guarded by isActive() so it's idempotent.
- Per-turn IPC uses a thread-local (reply/turn_state.py) instead of threading
  params through the whole engine chain: bridge sets broadcasting in, tool
  records the directive out; Tool.execute exposes broadcasting on ToolContext.

Bot typecheck clean; brain covered by tests/test_set_broadcast.py (+ existing
routing tests). Specs + docs updated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 10:08:30 +09:00

4.8 KiB
Raw Blame History

Real-time info modes (STREAM_BROWSER)

The bot answers via the Python brain (bridge/server.py -> src/jarvis). Real-time info is fetched by a tool the reply engine calls.

STREAM_BROWSER is the broadcast capability master flag; the live screen-share state then decides the search backend per turn:

  • STREAM_BROWSER=true (default): broadcasting is allowed.
    • Joining a voice channel auto-starts the broadcast.
    • While the broadcast is live: drive the on-screen Chrome (CDP at CDP_PORT, default 9222) to Google-search / play YouTube / read the page — visible on the Go-Live stream (VNC display :1).
    • While the broadcast is off: use Gemini (grounded search), no screen needed.
    • The user can toggle the broadcast on/off by voice ("방송 켜줘" / "방송 꺼줘"), routed through an LLM tool (no hardcoded phrase matching).
  • STREAM_BROWSER=false: broadcasting is disabled entirely. Any /stream or "start broadcast" request is refused, and real-time search always uses Gemini. Auth sub-mode is GEMINI_AUTH:
    • oauth (default): the Gemini CLI with a Google-account login (no API key).
    • apikey: the REST API keyed by GEMINI_API_KEY.

The brain receives the live broadcast state per request (ToolContext.broadcasting, threaded from the bot via the bridge); None means "no signal" (evals, text entry) and falls back to the master flag so behaviour is unchanged.

Components

Piece Path Status
Mode flag (bot) bot/src/config.ts screenBrowser, enforced in selfbot.ts done
Browser search core (Node/CDP) bot/scripts/stream-test/browse-search.mjs this change
Brain mode read src/jarvis/config.py stream_browser from env TODO
Gemini auth mode GEMINI_AUTH (oauth/apikey), GEMINI_API_KEY, GEMINI_MODEL (.env) + config.py done
browseAndSearch tool (true) src/jarvis/tools/builtin/browse_and_search.py -> subprocess the Node core TODO
Gemini search (false) realtime_search.py gemini_cli_search() (oauth, CLI) / gemini_search() (apikey, REST) done
Search routing by live broadcast web_search.py uses ToolContext.broadcasting + master flag done
Live broadcast state in tool ctx tools/base.py ToolContext.broadcasting from turn_state done
Per-turn state channel reply/turn_state.py (thread-local: broadcasting in, directive out) done
Bridge passes broadcast state bridge.ts ?broadcasting= + server.py think(broadcasting=) done
Auto-broadcast on voice join (true mode) bot/src/index.ts handleJoin() starts streamer + wires hooks done
Voice broadcast on/off toggle setBroadcast tool -> broadcast_action -> bot streamer.start/stop done
/stream + start refused (false mode) handleStream() gated + setBroadcast refuses done
Specs + docs/llm_contexts.md alongside each tool done

Design decisions

  • The browser tool (Python) subprocesses a Node script rather than adding a Python CDP/playwright dependency: the Node layer already owns Chrome/CDP (broadcast-helper.mjs, selfbot.ts), so the brain shells out to node browse-search.mjs <query> and wraps the JSON result in the engine's UNTRUSTED WEB EXTRACT envelope. Keeps the 39k-line Python brain dep-free.
  • Gemini has two auth sub-modes (GEMINI_AUTH):
    • oauth (default): shell out to the Gemini CLI (gemini -p <query> -o json --skip-trust, default approval mode — read-only tools like web search run headless, but write/shell tools are never auto-approved) authenticated by the user's Google account login. GEMINI_API_KEY/GOOGLE_API_KEY are stripped from the child env, and GOOGLE_GENAI_USE_GCA=true is set, so the CLI uses the account login (not API-key auth) and fails fast when no login exists rather than erroring on "no auth method". The CLI is resolved from PATH or ~/.local/bin/gemini; install with npm i -g @google/gemini-cli and sign in once via interactive gemini ("Sign in with Google").
    • apikey: the REST endpoint (generativelanguage.googleapis.com) via stdlib urllib with the google_search grounding tool - no SDK dependency.
  • Both Gemini paths and the browser path return the same ToolExecutionResult(success, reply_text) envelope, and are fail-open: any failure returns None and the caller degrades to the DDG/Brave/Wikipedia cascade.

Notes / verification

  • Grounded Gemini search needs real quota. On a free tier the grounded call can return HTTP 429 RESOURCE_EXHAUSTED (free-tier limit 0); the OAuth login must be a Google account with usable Gemini quota, otherwise the path 429s and fail-opens to DDG. The 30s subprocess timeout bounds the CLI's retry/backoff.
  • Behaviour covered by tests/test_realtime_gemini_cli.py.
Reference in New Issue View Git Blame Copy Permalink