2 Commits

Author SHA1 Message Date
javis-bot
35e754d6ee fix(docker): in-container Gemini CLI OAuth, broadcast audio, ports + hardening
Makes the all-in-one image actually run the new real-time-search features and
closes review gaps:
- Gemini OAuth path: install Node 22 + @google/gemini-cli (pinned 0.46.0) in the
  image; mount a DEDICATED host dir (~/.config/javis/gemini) holding only the
  OAuth creds to /root/.gemini (not the whole ~/.gemini). Verified in-container:
  `gemini -p ... -o json` returns a grounded answer with no API key.
- Broadcast audio: add PulseAudio + a headless null-sink (run-pulse.sh, new
  supervisor program); export XDG_RUNTIME_DIR/PULSE_SERVER so Chrome playback
  and the selfbot `ffmpeg -f pulse -i @DEFAULT_MONITOR@` share one daemon.
  Verified: default sink virtual_speaker, monitor present, ffmpeg capture OK.
- Bind the brain bridge to 127.0.0.1 only (internal, unauthenticated API).
- VNC host port is overridable; this server pins VNC_PORT=5902 (.env) since the
  host already runs Xvnc on 5901.

Verified in-container with CDI GPU passthrough: RTX 5050 visible, NVENC
encoders (h264/hevc/av1) available.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 01:36:30 +09:00
javis-bot
5d45d1d3bd feat(brain): route real-time search by live broadcast state
STREAM_BROWSER becomes the broadcast *capability* master flag; the live
screen-share state (new ToolContext.broadcasting, passed per turn by the bot)
decides the backend:
  - master off            -> broadcast disabled, always Gemini
  - master on + live on    -> on-screen Chrome (visible on the stream)
  - master on + live off   -> Gemini
context.broadcasting is None outside the voice path (evals, text entry, older
bot) and falls back to the master flag, so current behaviour is unchanged.
This is the brain-side foundation; bot-side wiring (bridge passes broadcast
state, auto-broadcast on voice join, voice on/off toggle) follows.

Specs + docs/llm_contexts.md updated. Covered by
tests/test_web_search_broadcast_routing.py.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 01:17:50 +09:00
11 changed files with 209 additions and 25 deletions

View File

@@ -19,6 +19,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
xfce4 xfce4-goodies dbus-x11 x11-utils xfonts-base \
fonts-noto-cjk fonts-noto-cjk-extra fonts-nanum \
ffmpeg tesseract-ocr \
pulseaudio pulseaudio-utils \
python3 python3-venv python3-pip \
novnc websockify supervisor gettext-base \
&& rm -rf /var/lib/apt/lists/*
@@ -31,6 +32,17 @@ RUN wget -q -O /tmp/chrome.deb https://dl.google.com/linux/direct/google-chrome-
# --- bun (Discord bot runtime/package manager) ---
RUN curl -fsSL https://bun.sh/install | bash
# --- Node.js + Gemini CLI (the GEMINI_AUTH=oauth real-time search path shells
# out to `gemini`; it needs a real Node runtime, not bun). The account
# login itself is provided at runtime via the mounted ~/.gemini (see
# docker-compose.yml); here we only install the binary. ---
ARG GEMINI_CLI_VERSION=0.46.0
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& npm install -g "@google/gemini-cli@${GEMINI_CLI_VERSION}" \
&& rm -rf /var/lib/apt/lists/* \
&& gemini --version
# --- Python brain/bridge deps (slim set) ---
COPY bridge/requirements-bridge.txt /app/bridge/requirements-bridge.txt
RUN python3 -m venv /opt/venv \

View File

@@ -69,14 +69,26 @@ services:
shm_size: "1gb" # Chrome needs a larger /dev/shm
ports:
# Host ports are overridable. If the HOST already runs VNC on 5901
# (see docs/vnc-xfce-setup.md), set VNC_PORT=5902 in .env.
# (see docs/vnc-xfce-setup.md), set VNC_PORT=5902 in .env — this server
# does, so .env pins VNC_PORT=5902.
- "${VNC_PORT:-5901}:5901" # VNC
- "${NOVNC_PORT:-6080}:6080" # noVNC (open in a browser)
- "${BRIDGE_PORT:-8765}:8765" # brain bridge (usually internal-only)
# Brain bridge: bind to loopback only — it is an internal bot<->brain API
# with no auth and must not be reachable off-host.
- "127.0.0.1:${BRIDGE_PORT:-8765}:8765"
volumes:
- javis_data:/data # jarvis db + memory
- whisper_cache:/root/.cache/huggingface # cached Whisper models
- piper_voices:/opt/piper-voices # TTS voices
# Gemini account login for GEMINI_AUTH=oauth real-time search. Mounts a
# DEDICATED dir holding only the Gemini OAuth creds (not the whole
# ~/.gemini), so the container can refresh its token without exposing
# unrelated host state. Seed it once with the host login:
# mkdir -p ~/.config/javis/gemini
# cp ~/.gemini/oauth_creds.json ~/.config/javis/gemini/
# Override GEMINI_OAUTH_DIR to point elsewhere. Only used when
# GEMINI_AUTH=oauth.
- ${GEMINI_OAUTH_DIR:-${HOME}/.config/javis/gemini}:/root/.gemini
volumes:
ollama_models:

View File

@@ -20,9 +20,15 @@ set -euo pipefail
: "${PIPER_VOICE_DIR:=/opt/piper-voices}"
: "${TTS_PIPER_MODEL_PATH:=${PIPER_VOICE_DIR}/${PIPER_VOICE}.onnx}"
# Audio: shared PulseAudio socket so Chrome (playback) and the selfbot ffmpeg
# (capture of @DEFAULT_MONITOR@) reach the same headless daemon (run-pulse.sh).
: "${XDG_RUNTIME_DIR:=/run/user/0}"
: "${PULSE_SERVER:=unix:${XDG_RUNTIME_DIR}/pulse/native}"
export VNC_RESOLUTION OLLAMA_BASE_URL OLLAMA_CHAT_MODEL OLLAMA_EMBED_MODEL \
WHISPER_MODEL WHISPER_DEVICE WHISPER_COMPUTE_TYPE JARVIS_DB_PATH \
PIPER_VOICE PIPER_VOICE_DIR TTS_PIPER_MODEL_PATH BRIDGE_HOST BRIDGE_PORT
PIPER_VOICE PIPER_VOICE_DIR TTS_PIPER_MODEL_PATH BRIDGE_HOST BRIDGE_PORT \
XDG_RUNTIME_DIR PULSE_SERVER
mkdir -p /data /app/config "$(dirname "$JARVIS_DB_PATH")"

23
docker/run-pulse.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# Headless PulseAudio for the broadcast audio path.
#
# The selfbot streamer captures desktop audio with `ffmpeg -f pulse -i
# @DEFAULT_MONITOR@` (bot/src/stream/selfbot.ts). In a container there is no
# real sound card, so we run a PulseAudio daemon with a null sink set as the
# default: Chrome/desktop play into it, and its `.monitor` source is what
# ffmpeg captures. Without this the broadcast fails to start when STREAM_AUDIO=1.
set -e
export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/run/user/0}"
mkdir -p "$XDG_RUNTIME_DIR/pulse"
chmod 700 "$XDG_RUNTIME_DIR"
exec pulseaudio \
--exit-idle-time=-1 \
--disallow-exit \
--disable-shm \
-n \
--load="module-native-protocol-unix auth-anonymous=1 socket=${XDG_RUNTIME_DIR}/pulse/native" \
--load="module-null-sink sink_name=virtual_speaker sink_properties=device.description=Virtual_Speaker" \
--load="module-always-sink" \
--log-target=stderr

View File

@@ -22,6 +22,15 @@ stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:pulse]
command=/app/docker/run-pulse.sh
priority=150
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:xfce]
command=/app/docker/run-xfce.sh
priority=200

View File

@@ -171,7 +171,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- **Weather** ([src/jarvis/tools/builtin/weather.py](src/jarvis/tools/builtin/weather.py), ~line 60) — `ollama_chat_model`, parses location/time/unit from the query.
- **Nutrition log_meal** ([src/jarvis/tools/builtin/nutrition/log_meal.py](src/jarvis/tools/builtin/nutrition/log_meal.py), lines 48 & 136) — `ollama_chat_model`, extracts nutrients, confirms logging.
- **Gemini real-time search** ([src/jarvis/tools/builtin/realtime_search.py](src/jarvis/tools/builtin/realtime_search.py)) — **external Gemini model**, NOT Ollama. Only on the `webSearch` route when `STREAM_BROWSER=false`; the sub-mode is `cfg.gemini_auth` (env `GEMINI_AUTH`, default `oauth`):
- **Gemini real-time search** ([src/jarvis/tools/builtin/realtime_search.py](src/jarvis/tools/builtin/realtime_search.py)) — **external Gemini model**, NOT Ollama. Used on the `webSearch` route whenever the on-screen Chrome path is NOT active: either `STREAM_BROWSER=false` (broadcast disabled) or `STREAM_BROWSER=true` with the live broadcast currently off (`context.broadcasting` False). Sub-mode is `cfg.gemini_auth` (env `GEMINI_AUTH`, default `oauth`):
- `oauth` (default) `gemini_cli_search()` — shells out to the Gemini CLI (`gemini -p <query> -o json --skip-trust`, default approval mode) authenticated by the user's Google-account login (`GEMINI_API_KEY`/`GOOGLE_API_KEY` stripped from the child env, `GOOGLE_GENAI_USE_GCA=true` set to select OAuth); model is whatever the CLI/account defaults to. Uses the CLI's built-in web-search grounding. Bounded by a 30s subprocess timeout.
- `apikey` `gemini_search()` — one REST `generateContent` call (`gemini_model`, default `gemini-2.0-flash`) with the `google_search` grounding tool; keyed by `GEMINI_API_KEY`.
Both return the fenced UNTRUSTED-WEB-EXTRACT envelope consumed by the main loop (#1). Fail-open: CLI missing / login expired / quota 429 / timeout / errors / missing key all fall through to the DDG cascade. The `STREAM_BROWSER=true` route (`browser_search()`) makes NO LLM call — it drives Chrome and scrapes Google results.

View File

@@ -1,16 +1,29 @@
# Real-time info modes (`STREAM_BROWSER`)
The bot answers via the Python brain (`bridge/server.py` -> `src/jarvis`). Real-time
info is fetched by a tool the reply engine calls. `STREAM_BROWSER` selects HOW:
info is fetched by a tool the reply engine calls.
- **true** (default): drive the on-screen Chrome (CDP at `CDP_PORT`, default 9222)
to Google-search / play YouTube / read the page. The action is visible on the
Go-Live broadcast. The browser is already up on the VNC display `:1`.
- **false**: use Gemini (grounded with Google Search) for real-time info. No
screen share needed (voice + Gemini only). Auth sub-mode is `GEMINI_AUTH`:
`STREAM_BROWSER` is the broadcast **capability** master flag; the **live**
screen-share state then decides the search backend per turn:
- **`STREAM_BROWSER=true`** (default): broadcasting is allowed.
- Joining a voice channel auto-starts the broadcast.
- While the broadcast is **live**: drive the on-screen Chrome (CDP at
`CDP_PORT`, default 9222) to Google-search / play YouTube / read the page —
visible on the Go-Live stream (VNC display `:1`).
- While the broadcast is **off**: use Gemini (grounded search), no screen needed.
- The user can toggle the broadcast on/off by voice ("방송 켜줘" / "방송 꺼줘"),
routed through an LLM tool (no hardcoded phrase matching).
- **`STREAM_BROWSER=false`**: broadcasting is disabled entirely. Any `/stream`
or "start broadcast" request is refused, and real-time search always uses
Gemini. Auth sub-mode is `GEMINI_AUTH`:
- `oauth` (default): the Gemini CLI with a Google-account login (no API key).
- `apikey`: the REST API keyed by `GEMINI_API_KEY`.
The brain receives the live broadcast state per request (`ToolContext.broadcasting`,
threaded from the bot via the bridge); `None` means "no signal" (evals, text
entry) and falls back to the master flag so behaviour is unchanged.
## Components
| Piece | Path | Status |
@@ -21,8 +34,13 @@ info is fetched by a tool the reply engine calls. `STREAM_BROWSER` selects HOW:
| Gemini auth mode | `GEMINI_AUTH` (oauth/apikey), `GEMINI_API_KEY`, `GEMINI_MODEL` (.env) + `config.py` | done |
| `browseAndSearch` tool (true) | `src/jarvis/tools/builtin/browse_and_search.py` -> subprocess the Node core | TODO |
| Gemini search (false) | `realtime_search.py` `gemini_cli_search()` (oauth, CLI) / `gemini_search()` (apikey, REST) | done |
| Registry (mode-gated) | `src/jarvis/tools/registry.py` `BUILTIN_TOOLS` | TODO |
| Specs + `docs/llm_contexts.md` | alongside each tool | TODO |
| Search routing by live broadcast | `web_search.py` uses `ToolContext.broadcasting` + master flag | done |
| Live broadcast state in tool ctx | `tools/base.py` `ToolContext.broadcasting` | done |
| Bridge passes broadcast state | bot `bridge.ts` request + `bridge/server.py` -> ToolContext | TODO |
| Auto-broadcast on voice join (true mode) | `bot/src/index.ts` `handleJoin()` starts streamer | TODO |
| Voice broadcast on/off toggle | brain tool -> reply directive -> bot `streamer.start/stop` | TODO |
| `/stream` + start refused (false mode) | `bot/src/index.ts` `handleStream()` already gated | done |
| Specs + `docs/llm_contexts.md` | alongside each tool | done |
## Design decisions

View File

@@ -22,6 +22,7 @@ class ToolContext:
max_retries: int,
user_print: Callable[[str], None],
language: Optional[str] = None,
broadcasting: Optional[bool] = None,
):
self.db = db
self.cfg = cfg
@@ -36,6 +37,13 @@ class ToolContext:
# treat absence as "no signal" and fall back to their own default
# rather than assuming English.
self.language = language
# Live broadcast (screen-share / Go-Live) state for THIS turn, passed in
# by the bot per request. Controls real-time search routing when the
# master flag ``cfg.stream_browser`` is on: broadcasting -> on-screen
# Chrome search (visible on the stream), not broadcasting -> Gemini.
# ``None`` means "no signal" (evals, text entry, older bot): callers
# fall back to the master flag so behaviour is unchanged.
self.broadcasting = broadcasting
class Tool(ABC):

View File

@@ -594,15 +594,27 @@ class WebSearchTool(Tool):
context.user_print(f"🌐 Searching the web for '{search_query}'")
debug_log(f" 🌐 searching for '{search_query}'", "web")
# Real-time info routing by STREAM_BROWSER (docs/stream_browser_modes.md):
# true -> drive the on-screen Chrome (visible on the broadcast),
# false -> Gemini grounded search. Either falls through to the
# DDG/Brave/Wikipedia cascade below if it yields nothing (fail-open).
# Real-time info routing (docs/stream_browser_modes.md):
# master flag cfg.stream_browser (env STREAM_BROWSER) is the
# broadcast *capability*; context.broadcasting is the *live*
# screen-share state for this turn (set by the bot).
# - master off -> broadcast disabled, always Gemini
# - master on + live on -> on-screen Chrome (visible on stream)
# - master on + live off -> Gemini
# context.broadcasting is None outside the voice path (evals, text
# entry, older bot) -> fall back to the master flag so behaviour is
# unchanged. Either backend falls through to the DDG/Brave/Wikipedia
# cascade below if it yields nothing (fail-open).
from .realtime_search import browser_search, gemini_search, gemini_cli_search
if getattr(cfg, "stream_browser", True):
master_browser = getattr(cfg, "stream_browser", True)
live = getattr(context, "broadcasting", None)
if live is None:
live = master_browser
use_browser = master_browser and live
if use_browser:
routed = browser_search(search_query)
if routed:
debug_log(" 🌐 routed via browser (STREAM_BROWSER=true)", "web")
debug_log(" 🌐 routed via browser (broadcast live)", "web")
return ToolExecutionResult(success=True, reply_text=routed)
elif getattr(cfg, "gemini_auth", "oauth") == "oauth":
routed = gemini_cli_search(search_query)

View File

@@ -7,14 +7,23 @@ memory.
### Real-time info routing (`STREAM_BROWSER`)
Before the DuckDuckGo cascade, `run()` routes by the env flag `STREAM_BROWSER`
(mirrored into `cfg.stream_browser`; see `docs/stream_browser_modes.md` and
`realtime_search.py`):
Before the DuckDuckGo cascade, `run()` routes by the broadcast capability flag
`cfg.stream_browser` (env `STREAM_BROWSER`) combined with the live broadcast
state `context.broadcasting` for the turn (see `docs/stream_browser_modes.md`
and `realtime_search.py`). `context.broadcasting` is `None` outside the voice
path (evals, text entry) and falls back to the master flag:
- **true** (default): `browser_search()` drives the on-screen Chrome (Node CDP
helper `bot/scripts/stream-test/browse-search.mjs`) to Google-search the
query, so the action is visible on the Go-Live broadcast.
- **false**: Gemini answers, with the sub-mode chosen by `cfg.gemini_auth`
| `cfg.stream_browser` | `context.broadcasting` | backend |
|---|---|---|
| True | True | on-screen Chrome (`browser_search()`) |
| True | False | Gemini |
| True | None | Chrome (no signal -> master flag) |
| False | any | Gemini (broadcast disabled, never Chrome) |
- **on-screen Chrome**: `browser_search()` drives Chrome (Node CDP helper
`bot/scripts/stream-test/browse-search.mjs`) to Google-search the query, so
the action is visible on the Go-Live broadcast.
- **Gemini**: answers, with the sub-mode chosen by `cfg.gemini_auth`
(env `GEMINI_AUTH`, default `oauth`):
- `oauth` (default): `gemini_cli_search()` shells out to the Gemini CLI
(`gemini -p <query> -o json --skip-trust`, default approval mode) using the

View File

@@ -0,0 +1,75 @@
"""Behaviour tests for real-time search routing by live broadcast state.
Routing contract (docs/stream_browser_modes.md):
- master flag cfg.stream_browser (env STREAM_BROWSER) = broadcast capability
- context.broadcasting = live screen-share state for the turn (set by the bot)
master live(ctx) -> backend
------- --------- -------
True True on-screen Chrome (browser_search)
True False Gemini
True None Chrome (no signal -> fall back to master)
False True/None Gemini (broadcast disabled, never Chrome)
"""
from types import SimpleNamespace
import jarvis.tools.builtin.realtime_search as rs
from jarvis.tools.base import ToolContext
from jarvis.tools.builtin.web_search import WebSearchTool
def _ctx(stream_browser, broadcasting):
cfg = SimpleNamespace(
web_search_enabled=True,
stream_browser=stream_browser,
gemini_auth="oauth",
gemini_api_key="",
gemini_model="gemini-2.0-flash",
)
return ToolContext(
db=None,
cfg=cfg,
system_prompt="",
original_prompt="",
redacted_text="",
max_retries=0,
user_print=lambda *a, **k: None,
language=None,
broadcasting=broadcasting,
)
def _route(monkeypatch, stream_browser, broadcasting):
"""Run webSearch and return which backend handled it ('browser'/'gemini')."""
monkeypatch.setattr(rs, "browser_search", lambda *a, **k: "BROWSER_RESULT")
monkeypatch.setattr(rs, "gemini_cli_search", lambda *a, **k: "GEMINI_RESULT")
monkeypatch.setattr(rs, "gemini_search", lambda *a, **k: "GEMINI_RESULT")
res = WebSearchTool().run({"search_query": "현재 서울 날씨"}, _ctx(stream_browser, broadcasting))
if res.reply_text == "BROWSER_RESULT":
return "browser"
if res.reply_text == "GEMINI_RESULT":
return "gemini"
return "fallthrough"
def test_master_on_live_on_uses_browser(monkeypatch):
assert _route(monkeypatch, True, True) == "browser"
def test_master_on_live_off_uses_gemini(monkeypatch):
assert _route(monkeypatch, True, False) == "gemini"
def test_master_on_no_signal_defaults_to_browser(monkeypatch):
# None = no live signal (evals / text entry / older bot) -> behave as master
assert _route(monkeypatch, True, None) == "browser"
def test_master_off_never_uses_browser_even_if_live(monkeypatch):
# STREAM_BROWSER=false disables broadcast entirely: always Gemini.
assert _route(monkeypatch, False, True) == "gemini"
def test_master_off_no_signal_uses_gemini(monkeypatch):
assert _route(monkeypatch, False, None) == "gemini"