tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

feat(brain): route real-time search by live broadcast state

Browse Source 
STREAM_BROWSER becomes the broadcast *capability* master flag; the live
screen-share state (new ToolContext.broadcasting, passed per turn by the bot)
decides the backend:
  - master off            -> broadcast disabled, always Gemini
  - master on + live on    -> on-screen Chrome (visible on the stream)
  - master on + live off   -> Gemini
context.broadcasting is None outside the voice path (evals, text entry, older
bot) and falls back to the master flag, so current behaviour is unchanged.
This is the brain-side foundation; bot-side wiring (bridge passes broadcast
state, auto-broadcast on voice join, voice on/off toggle) follows.

Specs + docs/llm_contexts.md updated. Covered by
tests/test_web_search_broadcast_routing.py.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-11 01:17:50 +09:00
parent f54e2a46ae
commit 5d45d1d3bd
6 changed files with 144 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/llm_contexts.md
View File

@@ -171,7 +171,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- **Weather** ([src/jarvis/tools/builtin/weather.py](src/jarvis/tools/builtin/weather.py), ~line 60) — `ollama_chat_model`, parses location/time/unit from the query.
- **Nutrition log_meal** ([src/jarvis/tools/builtin/nutrition/log_meal.py](src/jarvis/tools/builtin/nutrition/log_meal.py), lines 48 & 136) — `ollama_chat_model`, extracts nutrients, confirms logging.
- **Gemini real-time search** ([src/jarvis/tools/builtin/realtime_search.py](src/jarvis/tools/builtin/realtime_search.py)) — **external Gemini model**, NOT Ollama. Only on the `webSearch` route when `STREAM_BROWSER=false`; the sub-mode is `cfg.gemini_auth` (env `GEMINI_AUTH`, default `oauth`):
- **Gemini real-time search** ([src/jarvis/tools/builtin/realtime_search.py](src/jarvis/tools/builtin/realtime_search.py)) — **external Gemini model**, NOT Ollama. Used on the `webSearch` route whenever the on-screen Chrome path is NOT active: either `STREAM_BROWSER=false` (broadcast disabled) or `STREAM_BROWSER=true` with the live broadcast currently off (`context.broadcasting` False). Sub-mode is `cfg.gemini_auth` (env `GEMINI_AUTH`, default `oauth`):
- `oauth` (default) `gemini_cli_search()` — shells out to the Gemini CLI (`gemini -p <query> -o json --skip-trust`, default approval mode) authenticated by the user's Google-account login (`GEMINI_API_KEY`/`GOOGLE_API_KEY` stripped from the child env, `GOOGLE_GENAI_USE_GCA=true` set to select OAuth); model is whatever the CLI/account defaults to. Uses the CLI's built-in web-search grounding. Bounded by a 30s subprocess timeout.
- `apikey` `gemini_search()` — one REST `generateContent` call (`gemini_model`, default `gemini-2.0-flash`) with the `google_search` grounding tool; keyed by `GEMINI_API_KEY`.
Both return the fenced UNTRUSTED-WEB-EXTRACT envelope consumed by the main loop (#1). Fail-open: CLI missing / login expired / quota 429 / timeout / errors / missing key all fall through to the DDG cascade. The `STREAM_BROWSER=true` route (`browser_search()`) makes NO LLM call — it drives Chrome and scrapes Google results.

34
docs/stream_browser_modes.md
View File

@@ -1,16 +1,29 @@
# Real-time info modes (`STREAM_BROWSER`)
The bot answers via the Python brain (`bridge/server.py` -> `src/jarvis`). Real-time
info is fetched by a tool the reply engine calls. `STREAM_BROWSER` selects HOW:
info is fetched by a tool the reply engine calls.
- **true** (default): drive the on-screen Chrome (CDP at `CDP_PORT`, default 9222)
to Google-search / play YouTube / read the page. The action is visible on the
Go-Live broadcast. The browser is already up on the VNC display `:1`.
- **false**: use Gemini (grounded with Google Search) for real-time info. No
screen share needed (voice + Gemini only). Auth sub-mode is `GEMINI_AUTH`:
`STREAM_BROWSER` is the broadcast **capability** master flag; the **live**
screen-share state then decides the search backend per turn:
- **`STREAM_BROWSER=true`** (default): broadcasting is allowed.
- Joining a voice channel auto-starts the broadcast.
- While the broadcast is **live**: drive the on-screen Chrome (CDP at
`CDP_PORT`, default 9222) to Google-search / play YouTube / read the page —
visible on the Go-Live stream (VNC display `:1`).
- While the broadcast is **off**: use Gemini (grounded search), no screen needed.
- The user can toggle the broadcast on/off by voice ("방송 켜줘" / "방송 꺼줘"),
routed through an LLM tool (no hardcoded phrase matching).
- **`STREAM_BROWSER=false`**: broadcasting is disabled entirely. Any `/stream`
or "start broadcast" request is refused, and real-time search always uses
Gemini. Auth sub-mode is `GEMINI_AUTH`:
- `oauth` (default): the Gemini CLI with a Google-account login (no API key).
- `apikey`: the REST API keyed by `GEMINI_API_KEY`.
The brain receives the live broadcast state per request (`ToolContext.broadcasting`,
threaded from the bot via the bridge); `None` means "no signal" (evals, text
entry) and falls back to the master flag so behaviour is unchanged.
## Components
| Piece | Path | Status |
@@ -21,8 +34,13 @@ info is fetched by a tool the reply engine calls. `STREAM_BROWSER` selects HOW:
| Gemini auth mode | `GEMINI_AUTH` (oauth/apikey), `GEMINI_API_KEY`, `GEMINI_MODEL` (.env) + `config.py` | done |
| `browseAndSearch` tool (true) | `src/jarvis/tools/builtin/browse_and_search.py` -> subprocess the Node core | TODO |
| Gemini search (false) | `realtime_search.py` `gemini_cli_search()` (oauth, CLI) / `gemini_search()` (apikey, REST) | done |
| Registry (mode-gated) | `src/jarvis/tools/registry.py` `BUILTIN_TOOLS` | TODO |
| Specs + `docs/llm_contexts.md` | alongside each tool | TODO |
| Search routing by live broadcast | `web_search.py` uses `ToolContext.broadcasting` + master flag | done |
| Live broadcast state in tool ctx | `tools/base.py` `ToolContext.broadcasting` | done |
| Bridge passes broadcast state | bot `bridge.ts` request + `bridge/server.py` -> ToolContext | TODO |
| Auto-broadcast on voice join (true mode) | `bot/src/index.ts` `handleJoin()` starts streamer | TODO |
| Voice broadcast on/off toggle | brain tool -> reply directive -> bot `streamer.start/stop` | TODO |
| `/stream` + start refused (false mode) | `bot/src/index.ts` `handleStream()` already gated | done |
| Specs + `docs/llm_contexts.md` | alongside each tool | done |
## Design decisions