5b6a67963a
Some checks failed
Release / semantic-release (push) Successful in 25s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Release / build-linux (push) Failing after 7m8s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
OAuth cannot be done interactively in the headless container, so the login
must be seeded into the mounted ~/.gemini. Three problems are fixed:
- Mount fragility on the Windows Docker Desktop target: the creds mount
defaulted to ${HOME}/.config/javis/gemini, but ${HOME} is often unset when
compose runs outside a WSL shell, silently mounting the wrong dir. Default is
now the project-local ./docker/gemini-oauth (cross-platform), GEMINI_OAUTH_DIR
still overrides.
- No visibility: when oauth is selected but no login is seeded, the path
silently degraded to DDG/Brave. Added gemini_oauth_ready() + a one-time debug
hint and a startup entrypoint warning (skipped on the browser role, fail-open).
- Seeding guidance: oauth_creds.json is the essential credential (refresh token;
GOOGLE_GENAI_USE_GCA=true forces OAuth), which is what the readiness check and
warning verify; docs recommend copying the whole ~/.gemini for convenience.
Adds docker/gemini-oauth/ seed dir (.gitkeep) with the login files gitignored,
GEMINI_OAUTH_DIR in .env.example, and updates DEPLOY.md, stream_browser_modes.md
and llm_contexts.md. Covered by 3 new tests (10 passed total).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
81 lines
5.2 KiB
Markdown
81 lines
5.2 KiB
Markdown
# Real-time info modes (`STREAM_BROWSER`)
|
|
|
|
The bot answers via the Python brain (`bridge/server.py` -> `src/jarvis`). Real-time
|
|
info is fetched by a tool the reply engine calls.
|
|
|
|
`STREAM_BROWSER` is the broadcast **capability** master flag; the **live**
|
|
screen-share state then decides the search backend per turn:
|
|
|
|
- **`STREAM_BROWSER=true`** (default): broadcasting is allowed.
|
|
- Joining a voice channel auto-starts the broadcast.
|
|
- While the broadcast is **live**: drive the on-screen Chrome (CDP at
|
|
`CDP_PORT`, default 9222) to Google-search / play YouTube / read the page —
|
|
visible on the Go-Live stream (VNC display `:1`).
|
|
- While the broadcast is **off**: use Gemini (grounded search), no screen needed.
|
|
- The user can toggle the broadcast on/off by voice ("방송 켜줘" / "방송 꺼줘"),
|
|
routed through an LLM tool (no hardcoded phrase matching).
|
|
- **`STREAM_BROWSER=false`**: broadcasting is disabled entirely. Any `/stream`
|
|
or "start broadcast" request is refused, and real-time search always uses
|
|
Gemini. Auth sub-mode is `GEMINI_AUTH`:
|
|
- `oauth` (default): the Gemini CLI with a Google-account login (no API key).
|
|
- `apikey`: the REST API keyed by `GEMINI_API_KEY`.
|
|
|
|
The brain receives the live broadcast state per request (`ToolContext.broadcasting`,
|
|
threaded from the bot via the bridge); `None` means "no signal" (evals, text
|
|
entry) and falls back to the master flag so behaviour is unchanged.
|
|
|
|
## Components
|
|
|
|
| Piece | Path | Status |
|
|
|---|---|---|
|
|
| Mode flag (bot) | `bot/src/config.ts` `screenBrowser`, enforced in `selfbot.ts` | done |
|
|
| Browser search core (Node/CDP) | `bot/scripts/stream-test/browse-search.mjs` | this change |
|
|
| Brain mode read | `src/jarvis/config.py` `stream_browser` from env | TODO |
|
|
| Gemini auth mode | `GEMINI_AUTH` (oauth/apikey), `GEMINI_API_KEY`, `GEMINI_MODEL` (.env) + `config.py` | done |
|
|
| `browseAndSearch` tool (true) | `src/jarvis/tools/builtin/browse_and_search.py` -> subprocess the Node core | TODO |
|
|
| Gemini search (false) | `realtime_search.py` `gemini_cli_search()` (oauth, CLI) / `gemini_search()` (apikey, REST) | done |
|
|
| Search routing by live broadcast | `web_search.py` uses `ToolContext.broadcasting` + master flag | done |
|
|
| Live broadcast state in tool ctx | `tools/base.py` `ToolContext.broadcasting` from `turn_state` | done |
|
|
| Per-turn state channel | `reply/turn_state.py` (thread-local: broadcasting in, directive out) | done |
|
|
| Bridge passes broadcast state | `bridge.ts` `?broadcasting=` + `server.py` `think(broadcasting=)` | done |
|
|
| Auto-broadcast on voice join (true mode) | `bot/src/index.ts` `handleJoin()` starts streamer + wires hooks | done |
|
|
| Voice broadcast on/off toggle | `setBroadcast` tool -> `broadcast_action` -> bot `streamer.start/stop` | done |
|
|
| `/stream` + start refused (false mode) | `handleStream()` gated + `setBroadcast` refuses | done |
|
|
| Specs + `docs/llm_contexts.md` | alongside each tool | done |
|
|
|
|
## Design decisions
|
|
|
|
- The browser tool (Python) **subprocesses a Node script** rather than adding a
|
|
Python CDP/playwright dependency: the Node layer already owns Chrome/CDP
|
|
(`broadcast-helper.mjs`, `selfbot.ts`), so the brain shells out to
|
|
`node browse-search.mjs <query>` and wraps the JSON result in the engine's
|
|
`UNTRUSTED WEB EXTRACT` envelope. Keeps the 39k-line Python brain dep-free.
|
|
- Gemini has two auth sub-modes (`GEMINI_AUTH`):
|
|
- `oauth` (default): shell out to the Gemini CLI (`gemini -p <query> -o json
|
|
--skip-trust`, default approval mode — read-only tools like web search run
|
|
headless, but write/shell tools are never auto-approved) authenticated by
|
|
the user's Google account login. `GEMINI_API_KEY`/`GOOGLE_API_KEY` are stripped from the child
|
|
env, and `GOOGLE_GENAI_USE_GCA=true` is set, so the CLI uses the account
|
|
login (not API-key auth) and fails fast when no login exists rather than
|
|
erroring on "no auth method". The CLI is resolved from `PATH` or
|
|
`~/.local/bin/gemini`; install with `npm i -g @google/gemini-cli` and sign
|
|
in once via interactive `gemini` ("Sign in with Google"). In Docker the login
|
|
can't be done interactively in the headless container: seed it instead by
|
|
copying a logged-in `~/.gemini` into the project-local `docker/gemini-oauth`
|
|
bind mount (or set `GEMINI_OAUTH_DIR`); the container reads/refreshes the
|
|
token there. `gemini_oauth_ready()` gates an actionable log hint, and the
|
|
entrypoint warns on startup, when oauth is selected but no login is seeded.
|
|
- `apikey`: the REST endpoint (`generativelanguage.googleapis.com`) via stdlib
|
|
`urllib` with the `google_search` grounding tool - no SDK dependency.
|
|
- Both Gemini paths and the browser path return the same
|
|
`ToolExecutionResult(success, reply_text)` envelope, and are fail-open: any
|
|
failure returns `None` and the caller degrades to the DDG/Brave/Wikipedia
|
|
cascade.
|
|
|
|
## Notes / verification
|
|
- Grounded Gemini search needs real quota. On a free tier the grounded call can
|
|
return HTTP 429 `RESOURCE_EXHAUSTED` (free-tier limit 0); the OAuth login must
|
|
be a Google account with usable Gemini quota, otherwise the path 429s and
|
|
fail-opens to DDG. The 30s subprocess timeout bounds the CLI's retry/backoff.
|
|
- Behaviour covered by `tests/test_realtime_gemini_cli.py`.
|