javis_bot/docs/stream_browser_modes.md

Real-time info modes (STREAM_BROWSER)

The bot answers via the Python brain (bridge/server.py -> src/jarvis). Real-time info is fetched by a tool the reply engine calls. STREAM_BROWSER selects HOW:

• true (default): drive the on-screen Chrome (CDP at CDP_PORT, default 9222) to Google-search / play YouTube / read the page. The action is visible on the Go-Live broadcast. The browser is already up on the VNC display :1.
• false: use Gemini (grounded with Google Search) for real-time info. No screen share needed (voice + Gemini only). Auth sub-mode is GEMINI_AUTH:
  • oauth (default): the Gemini CLI with a Google-account login (no API key).
  • apikey: the REST API keyed by GEMINI_API_KEY.

Components

Piece	Path	Status
Mode flag (bot)	bot/src/config.ts screenBrowser, enforced in selfbot.ts	done
Browser search core (Node/CDP)	bot/scripts/stream-test/browse-search.mjs	this change
Brain mode read	src/jarvis/config.py stream_browser from env	TODO
Gemini auth mode	GEMINI_AUTH (oauth/apikey), GEMINI_API_KEY, GEMINI_MODEL (.env) + config.py	done
browseAndSearch tool (true)	src/jarvis/tools/builtin/browse_and_search.py -> subprocess the Node core	TODO
Gemini search (false)	realtime_search.py gemini_cli_search() (oauth, CLI) / gemini_search() (apikey, REST)	done
Registry (mode-gated)	src/jarvis/tools/registry.py BUILTIN_TOOLS	TODO
Specs + docs/llm_contexts.md	alongside each tool	TODO

Design decisions

• The browser tool (Python) subprocesses a Node script rather than adding a Python CDP/playwright dependency: the Node layer already owns Chrome/CDP (broadcast-helper.mjs, selfbot.ts), so the brain shells out to node browse-search.mjs <query> and wraps the JSON result in the engine's UNTRUSTED WEB EXTRACT envelope. Keeps the 39k-line Python brain dep-free.
• Gemini has two auth sub-modes (GEMINI_AUTH):
  • oauth (default): shell out to the Gemini CLI (gemini -p <query> -o json --skip-trust, default approval mode — read-only tools like web search run headless, but write/shell tools are never auto-approved) authenticated by the user's Google account login. GEMINI_API_KEY/GOOGLE_API_KEY are stripped from the child env, and GOOGLE_GENAI_USE_GCA=true is set, so the CLI uses the account login (not API-key auth) and fails fast when no login exists rather than erroring on "no auth method". The CLI is resolved from PATH or ~/.local/bin/gemini; install with npm i -g @google/gemini-cli and sign in once via interactive gemini ("Sign in with Google").
  • apikey: the REST endpoint (generativelanguage.googleapis.com) via stdlib urllib with the google_search grounding tool - no SDK dependency.
• Both Gemini paths and the browser path return the same ToolExecutionResult(success, reply_text) envelope, and are fail-open: any failure returns None and the caller degrades to the DDG/Brave/Wikipedia cascade.

Notes / verification

• Grounded Gemini search needs real quota. On a free tier the grounded call can return HTTP 429 RESOURCE_EXHAUSTED (free-tier limit 0); the OAuth login must be a Google account with usable Gemini quota, otherwise the path 429s and fail-opens to DDG. The 30s subprocess timeout bounds the CLI's retry/backoff.
• Behaviour covered by tests/test_realtime_gemini_cli.py.