DuckDuckGo now serves only its bot-check page (game PC and host alike), so every search fell to Wikipedia full-text search, which picked unrelated pages (박진영 for songs for a rainy day, 대한민국 for a graphics card ranking). Live 2026-11-18 answers took 9-15 s and were wrong. - Exa's free MCP endpoint (no key) is the first provider after the live-broadcast Chrome; it returns page text, so no fetch is needed. - Gemini runs only when Exa found nothing and a CLI login (oauth) or key (apikey) is present; without a login the CLI is no longer started (about 2 s per search). - Wikipedia is used only when the page title is named in the query; otherwise the honest nothing-found envelope. - Letters of a script the user did not use are dropped from the query (the model searched '사과와 배 중哪个가 더 단가요'). - DuckDuckGo removed; spec, docs and tests follow the new order. - weather: any forecast failure (not only requests errors) tries met.no. Real model (qwen3.6-35b-a3b, game PC), real network: graphics card ranking, apples vs pears, rainy-day genres and black holes all answered correctly in 3.7-5.2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.3 KiB
5.3 KiB
Real-time info modes (STREAM_BROWSER)
The bot answers via the Python brain (bridge/server.py -> src/jarvis). Real-time
info is fetched by a tool the reply engine calls.
STREAM_BROWSER is the broadcast capability master flag; the live
screen-share state then decides the search backend per turn:
STREAM_BROWSER=true(default): broadcasting is allowed.- Joining a voice channel auto-starts the broadcast.
- While the broadcast is live: drive the on-screen Chrome (CDP at
CDP_PORT, default 9222) to Google-search / play YouTube / read the page — visible on the Go-Live stream (VNC display:1). - While the broadcast is off: the normal webSearch chain (Exa, then Gemini, then
Brave and Wikipedia;
src/jarvis/tools/builtin/web_search.spec.md), no screen needed. - The user can toggle the broadcast on/off by voice ("방송 켜줘" / "방송 꺼줘"), routed through an LLM tool (no hardcoded phrase matching).
STREAM_BROWSER=false: broadcasting is disabled entirely. Any/streamor "start broadcast" request is refused, and real-time search always uses the normal webSearch chain. Gemini's auth sub-mode there isGEMINI_AUTH:oauth(default): the Gemini CLI with a Google-account login (no API key).apikey: the REST API keyed byGEMINI_API_KEY.
The brain receives the live broadcast state per request (ToolContext.broadcasting,
threaded from the bot via the bridge); None means "no signal" (evals, text
entry) and falls back to the master flag so behaviour is unchanged.
Components
| Piece | Path | Status |
|---|---|---|
| Mode flag (bot) | bot/src/config.ts screenBrowser, enforced in selfbot.ts |
done |
| Browser search core (Node/CDP) | bot/scripts/stream-test/browse-search.mjs |
this change |
| Brain mode read | src/jarvis/config.py stream_browser from env |
TODO |
| Gemini auth mode | GEMINI_AUTH (oauth/apikey), GEMINI_API_KEY, GEMINI_MODEL (.env) + config.py |
done |
browseAndSearch tool (true) |
src/jarvis/tools/builtin/browse_and_search.py -> subprocess the Node core |
TODO |
| Gemini search (false) | realtime_search.py gemini_cli_search() (oauth, CLI) / gemini_search() (apikey, REST) |
done |
| Search routing by live broadcast | web_search.py uses ToolContext.broadcasting + master flag |
done |
| Live broadcast state in tool ctx | tools/base.py ToolContext.broadcasting from turn_state |
done |
| Per-turn state channel | reply/turn_state.py (thread-local: broadcasting in, directive out) |
done |
| Bridge passes broadcast state | bridge.ts ?broadcasting= + server.py think(broadcasting=) |
done |
| Auto-broadcast on voice join (true mode) | bot/src/index.ts handleJoin() starts streamer + wires hooks |
done |
| Voice broadcast on/off toggle | setBroadcast tool -> broadcast_action -> bot streamer.start/stop |
done |
/stream + start refused (false mode) |
handleStream() gated + setBroadcast refuses |
done |
Specs + docs/llm_contexts.md |
alongside each tool | done |
Design decisions
- The browser tool (Python) subprocesses a Node script rather than adding a
Python CDP/playwright dependency: the Node layer already owns Chrome/CDP
(
broadcast-helper.mjs,selfbot.ts), so the brain shells out tonode browse-search.mjs <query>and wraps the JSON result in the engine'sUNTRUSTED WEB EXTRACTenvelope. Keeps the 39k-line Python brain dep-free. - Gemini has two auth sub-modes (
GEMINI_AUTH):oauth(default): shell out to the Gemini CLI (gemini -p <query> -o json --skip-trust, default approval mode — read-only tools like web search run headless, but write/shell tools are never auto-approved) authenticated by the user's Google account login.GEMINI_API_KEY/GOOGLE_API_KEYare stripped from the child env, andGOOGLE_GENAI_USE_GCA=trueis set, so the CLI uses the account login (not API-key auth) and fails fast when no login exists rather than erroring on "no auth method". The CLI is resolved fromPATHor~/.local/bin/gemini; install withnpm i -g @google/gemini-cliand sign in once via interactivegemini("Sign in with Google"). In Docker the login can't be done interactively in the headless container: seed it instead by copying a logged-in~/.geminiinto the project-localdocker/gemini-oauthbind mount (or setGEMINI_OAUTH_DIR); the container reads/refreshes the token there.gemini_oauth_ready()gates an actionable log hint, and the entrypoint warns on startup, when oauth is selected but no login is seeded.apikey: the REST endpoint (generativelanguage.googleapis.com) via stdliburllibwith thegoogle_searchgrounding tool - no SDK dependency.
- Both Gemini paths and the browser path return the same
ToolExecutionResult(success, reply_text)envelope, and are fail-open: any failure returnsNoneand the caller goes on down the webSearch chain.
Notes / verification
- Grounded Gemini search needs real quota. On a free tier the grounded call can
return HTTP 429
RESOURCE_EXHAUSTED(free-tier limit 0); the OAuth login must be a Google account with usable Gemini quota, otherwise the path 429s and fail-opens to Brave/Wikipedia. The 30s subprocess timeout bounds the CLI's retry/backoff. - Behaviour covered by
tests/test_realtime_gemini_cli.py.