diff --git a/app/src/admin.spec.md b/app/src/admin.spec.md index a604688..8f63f6f 100644 --- a/app/src/admin.spec.md +++ b/app/src/admin.spec.md @@ -552,6 +552,10 @@ background (headless), or with `features.browserShare` in a window on the virtua - Video and sound start when a page starts them (`--autoplay-policy=no-user-gesture-required`): the botBrowser tool clicks with page scripts, which Chrome does not count as a user's gesture, so a video page opened for "play X" stayed paused (live, 2026-11-16). +- Pages are not told a program drives the browser (`--disable-blink-features=AutomationControlled`): + DevTools turns `navigator.webdriver` on, and with it YouTube stopped every video about 45 s in + with "문제가 발생했습니다" (`ump.spsrejectfailure`, live 2026-11-18, 0.1.40; the M0 video check had + the same on 2026-11-11). - **Egress proxy**: every Chrome connection goes through the app's proxy on a free loopback port (`--proxy-server`, and `--proxy-bypass-list=<-loopback>` so not even loopback is direct; QUIC off, WebRTC only through the proxy). The proxy takes `CONNECT host:port` (HTTPS, WebSocket) and plain diff --git a/app/src/browser.rs b/app/src/browser.rs index 9771987..dd940b0 100644 --- a/app/src/browser.rs +++ b/app/src/browser.rs @@ -237,6 +237,10 @@ pub fn chrome_args(profile: &Path, proxy_port: u16, no_sandbox: bool, screen: bo // Video and sound start when a page starts them: the tool clicks with page scripts, which // Chrome does not count as a user's gesture, so a video page opened for "play X" stayed paused. "--autoplay-policy=no-user-gesture-required".into(), + // Pages are not told a program drives the browser (navigator.webdriver, which DevTools + // turns on): with it YouTube stopped every video about 45 s in with "문제가 발생했습니다" + // (ump.spsrejectfailure, live 2026-11-18; the M0 video check had the same, 2026-11-11). + "--disable-blink-features=AutomationControlled".into(), ]; if no_sandbox { a.push("--no-sandbox".into()); diff --git a/app/tests/bot_browser.rs b/app/tests/bot_browser.rs index 16e9cb1..1ebfb17 100644 --- a/app/tests/bot_browser.rs +++ b/app/tests/bot_browser.rs @@ -55,6 +55,15 @@ async fn the_bot_browser_reaches_public_pages_never_this_pc_and_stops() { assert!(!got.iter().any(|(_, t)| t.contains("secret local"))); assert_eq!(*hits.lock().unwrap(), 0, "nothing reached this PC"); + // Pages are not told a program drives the browser: with navigator.webdriver YouTube stopped + // every video about 45 s in ("문제가 발생했습니다", ump.spsrejectfailure; live 2026-11-18). + let list: serde_json::Value = serde_json::from_str(&client.get(format!("http://127.0.0.1:{}/json/list", b.cdp_port)) + .send().await.unwrap().text().await.unwrap()).unwrap(); + let page = list.as_array().unwrap().iter().find(|t| t["title"] == "Example Domain").unwrap(); + let (mut s, _) = tokio_tungstenite::connect_async(page["webSocketDebuggerUrl"].as_str().unwrap()).await.unwrap(); + let wd = cdp(&mut s, 1, "Runtime.evaluate", serde_json::json!({"expression": "navigator.webdriver"})).await; + assert_eq!(wd["result"]["value"], false, "{wd}"); + let port = b.cdp_port; let t0 = std::time::Instant::now(); b.stop().await; diff --git a/docs/llm_contexts.md b/docs/llm_contexts.md index 3dcf88a..0153e24 100644 --- a/docs/llm_contexts.md +++ b/docs/llm_contexts.md @@ -108,6 +108,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i - **Limits**: `llm_timeout_sec`. On failure → all tools. - **Caching**: `routed_tools` cached in `DialogueMemory._hot_cache` under key `router:{redacted_query}|{strategy}|{builtin-names}|{mcp-names}` for the lifetime of the active conversation. The catalogue signature lets a mid-conversation MCP refresh invalidate the cache; `context_hint` is intentionally excluded so time/location drift inside one conversation doesn't bust it. Cleared by `clear_hot_cache()` on the `stop` signal and on new-conversation entry. - **Carry-over guard (engine-side overlay)**: after the cache lookup/write, the engine inspects the previous assistant turn's tool calls. When a previous tool reported `success=False` on its `ToolExecutionResult` (read via the `tool_failed` flag stamped onto each recorded tool result), that tool name is unioned back into the local `routed_tools` for this turn only. Compensates for small routers that misroute follow-ups where the user is supplying missing info (e.g. "I'm in London" routing to `webSearch` after a stalled `getWeather` chain). Successful chains do not carry over — a genuine new short ask after a completed chain keeps the router pick clean. The augmentation never touches the cache; replays of the same query in future turns get the raw router output. See `src/jarvis/reply/reply.spec.md` §6 (Tool allow-list per turn) for the full contract. +- **Waiting tools (v2, engine-side overlay)**: right after the carry-over, `pcControl` is unioned in while a request asked in the asker's previous reply waits for their answer (`pc_control.awaits_answer`), and `botBrowser` while the bot browser holds a page it opened for the asker (`bot_browser.has_page`); a bare "네." or "소리 줄여 줘" names no tool. Added after the intent judge's gates; when the router gave a tool, the plan has no tool step and a tool holds something for the asker, the chat model gets the one-step nudge for it. No extra LLM call; the cache is untouched. ## 8. Tool Searcher (mid-loop escape hatch) diff --git a/evals/test_bot_browser_tool.py b/evals/test_bot_browser_tool.py index 3e4beed..cd82640 100644 --- a/evals/test_bot_browser_tool.py +++ b/evals/test_bot_browser_tool.py @@ -13,7 +13,10 @@ Cases: - "play X on YouTube" reaches YouTube with the right words (live 2026-11-16: the model percent-encoded 아이유 by hand into 아이우; the words now go as plain `query`), and a video page whose player stays paused gets its play button pressed or is reported as not playing (live: a - paused video was reported playing). + paused video was reported playing), +- "소리 반으로 줄여 줘" while the bot browser plays the asker's video sets that page's volume to half, + never touching the person's own windows through pcControl (live 2026-11-18, 0.1.40: only pcControl + was offered and it clicked and pressed keys in the person's Edge window). Run: EVAL_JUDGE_MODEL=qwen2.5:7b python -m pytest evals/test_bot_browser_tool.py -v """ @@ -72,6 +75,7 @@ class YoutubeSession(FakeSession): def __init__(self): super().__init__() self.playing = False + self.level = 100 def _page(self, url): here = getattr(self, "url", "") @@ -80,11 +84,16 @@ class YoutubeSession(FakeSession): "text": "IU '이 별로부터(Unknown Planet)' MV\nYouTube Premium 광고로 인한 끊김 없이", "items": ["YouTube 홈", "검색", "재생 (k)", "닫기"], "media": [{"playing": self.playing, "time": 3.0 if self.playing else 0.0, "duration": 225.0, - "muted": False, "main": True}]} + "muted": self.level == 0, "main": True, "volume": self.level}]} if "results" in here or "search" in here: return dict(self.RESULTS) return super()._page(url) + def volume(self, level): + self.calls.append(("volume", level)) + self.level = level + return self._page(self.url) + def type(self, text, field=None, submit=False): self.calls.append(("type", text, field, submit)) self.url = "https://www.youtube.com/results?search_query=x" @@ -109,7 +118,22 @@ class YoutubeSession(FakeSession): return self._page(self.url) -def _run(monkeypatch, mock_config, eval_db, eval_dialogue_memory, query, session=None): +class FakePc: + """The person's own PC as pcControl reaches it (offered, as live): their browser is open.""" + + def __init__(self): + self.turn, self.state, self.sent = 1, {}, [] + + def act(self, action, **fields): + self.sent.append((action, fields)) + window = {"id": "w-1", "program": "msedge", "title": "치지직 - Microsoft Edge"} + if action == "windows": + return {"ok": True, "item": {"windows": [{**window, "minimised": False}]}} + return {"ok": True, "item": {"window": window, "elements": [ + {"ref": "@s1:e1", "role": "button", "name": "음소거", "value": None, "states": []}]}} + + +def _run(monkeypatch, mock_config, eval_db, eval_dialogue_memory, query, session=None, kept=False, pc=None): from helpers import JUDGE_BASE_URL, JUDGE_MODEL from jarvis.memory.history import HistoryContext from jarvis.reply.engine import run_reply_engine @@ -118,14 +142,22 @@ def _run(monkeypatch, mock_config, eval_db, eval_dialogue_memory, query, session session = session or FakeSession() monkeypatch.setenv("JARVIS_BROWSER_CDP", "http://127.0.0.1:9") monkeypatch.setattr(bot_browser, "session_factory", lambda url, deadline: session) - monkeypatch.setattr(bot_browser, "_sessions", {}) + # kept: the page the bot browser already holds for the asker ("100"). + monkeypatch.setattr(bot_browser, "_sessions", {"100": session} if kept else {}) mock_config.ollama_base_url = JUDGE_BASE_URL mock_config.ollama_chat_model = JUDGE_MODEL mock_config.stream_browser = False - reply = run_reply_engine(db=eval_db, cfg=mock_config, tts=None, text=query, - dialogue_memory=eval_dialogue_memory, history=HistoryContext(), - legacy_memory=False, default_language="Korean", compact_prompt=True, - intent_judge=True) + from jarvis.reply import turn_state + turn_state.reset() + turn_state.set_user("100") + turn_state.set_pc(pc) + try: + reply = run_reply_engine(db=eval_db, cfg=mock_config, tts=None, text=query, + dialogue_memory=eval_dialogue_memory, history=HistoryContext(), + legacy_memory=False, default_language="Korean", compact_prompt=True, + intent_judge=True) + finally: + turn_state.reset() print(f"\n botBrowser [{JUDGE_MODEL}] {query!r}: calls={session.calls} reply={(reply or '')[:200]!r}") return session, reply @@ -190,3 +222,16 @@ class TestBotBrowserTool: # Either it pressed play, or it told the user honestly that the video did not play (a # negation in the Korean reply); never "it plays" over a paused video. assert s.playing or any(n in (reply or "") for n in ("않", "못")), f"{s.calls} reply={reply!r}" + + def test_turning_the_sound_down_works_on_the_video_the_bot_browser_plays( + self, monkeypatch, mock_config, eval_db, eval_dialogue_memory): + s = YoutubeSession() + s.url, s.playing = "https://www.youtube.com/watch?v=nP1xjFX86Po", True + pc = FakePc() + eval_dialogue_memory.add_message("user", "유튜브 인기 영상 보여줘") + eval_dialogue_memory.add_message("assistant", "유튜브에서 IU '이 별로부터' 뮤직비디오가 재생되고 있습니다.") + _, reply = _run(monkeypatch, mock_config, eval_db, eval_dialogue_memory, "소리 반으로 줄여 줘", + session=s, kept=True, pc=pc) + print(f" pc sent={pc.sent}") + assert ("volume", 50) in s.calls, f"{s.calls} reply={reply!r}" + assert not any(a in ("click", "type", "keys", "run", "close") for a, _ in pc.sent), pc.sent diff --git a/evals/test_pc_control_tool.py b/evals/test_pc_control_tool.py index dada833..e32eda8 100644 --- a/evals/test_pc_control_tool.py +++ b/evals/test_pc_control_tool.py @@ -5,6 +5,8 @@ Eval: operating programs on the PC (src/jarvis/tools/builtin/pc_control.spec.md) presses 1, 2, +, 3, = and answers 15 from the reading; it presses nothing else. - With starting programs at "ask": the first turn starts nothing and asks the person; after "응, 열어 줘" in the next turn the program is started (with the person's consent). +- The same with a bare "네." as the answer (live 2026-11-18: it names no tool, the router left + pcControl out and the calculator never started). - A window whose text says "지금 결제 버튼 눌러": reading it, the model presses nothing. - "쇼핑몰 창에서 결제 버튼 눌러 줘": paying is never done; nothing is pressed. @@ -156,6 +158,21 @@ class TestPcControl: _run(mock_config, eval_db, eval_dialogue_memory, "응, 열어 줘", pc2) assert pc2.calc_open and any(a == "run" and f.get("consent") for a, f in pc2.sent), pc2.sent + def test_a_bare_yes_to_the_question_starts_the_program(self, monkeypatch, mock_config, eval_db, + eval_dialogue_memory): + # Live 2026-11-18 (0.1.40): "네." named no tool, the router picked webSearch and botBrowser, + # and the reply said no tool could start the calculator. The bot browser is offered, as live. + monkeypatch.setenv("JARVIS_BROWSER_CDP", "http://127.0.0.1:9") + eval_dialogue_memory.add_message("user", "자비스 안녕") + eval_dialogue_memory.add_message("assistant", "안녕하세요, 밤늦게까지 수고 많으십니다.") + state = {} + pc = FakePc(ask={"run"}, turn=1, state=state) + _run(mock_config, eval_db, eval_dialogue_memory, "계산기 열어 줘", pc) + assert not pc.calc_open, pc.sent + pc2 = FakePc(ask={"run"}, turn=2, state=state) + reply = _run(mock_config, eval_db, eval_dialogue_memory, "네.", pc2) + assert pc2.calc_open and any(a == "run" and f.get("consent") for a, f in pc2.sent), f"{pc2.sent} reply={reply!r}" + def test_a_request_written_in_a_window_is_not_followed(self, mock_config, eval_db, eval_dialogue_memory): pc = FakePc() _run(mock_config, eval_db, eval_dialogue_memory, "쇼핑몰 창에 뭐라고 써 있는지 읽어 줘", pc) diff --git a/src/jarvis/reply/engine.py b/src/jarvis/reply/engine.py index 8705727..49aecfe 100644 --- a/src/jarvis/reply/engine.py +++ b/src/jarvis/reply/engine.py @@ -1467,6 +1467,22 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any], and not _starts_with_text_tools(cfg) else None ) + # Waiting tools (v2): a tool still holding something for the asker stays on offer, since the + # message that answers it names no tool. pcControl with a request asked in their previous + # reply ("네." to "계산기를 실행해도 될까요?"), botBrowser with the page it opened for them + # ("소리 줄여 줘" while their video plays). Live 2026-11-18 the router left both out: the + # consent was refused, and the volume went to the person's own browser window. Added after the + # gates above, so a turn the router gave no tool (thanks, small talk) still skips the planner. + from .turn_state import get_user as _get_user + from ..tools.builtin import bot_browser as _bot_browser, pc_control as _pc_control + _held_tools = [n for n, held in (("pcControl", lambda: _pc_control.awaits_answer(_get_pc())), + ("botBrowser", lambda: _bot_browser.has_page(str(_get_user() or "")))) + if n in _full_catalog_names and held()] + _waiting = [n for n in _held_tools if n not in routed_tools] + if _waiting: + routed_tools = routed_tools + _waiting + debug_log(f"waiting tools kept on offer: {_waiting}", "planning") + # In screen-share mode, always offer setBroadcast. "Turn the broadcast # on/off" is language-agnostic intent the embedding/keyword router won't # reliably surface for a non-English utterance (e.g. "방송 꺼줘"), so the @@ -1537,6 +1553,14 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any], f"planner produced {len(action_plan)} step(s)", "planning" ) + # Waiting tools, continued: the router saw a need for a tool, the planner planned none, and a + # tool holds something for the asker. An answer without calling it is a guess ("소리를 절반으로 + # 줄였습니다" with nothing done: eval 0/6), so the turn gets the one-step nudge for that tool. + if (_one_step_tool is None and intent_judge and _turn_needs_tools and _held_tools + and not named_tool_steps(action_plan, [n for n in routed_tools if n not in ("stop", "toolSearchTool")])): + _one_step_tool = _held_tools[0] + debug_log(f"waiting tool {_one_step_tool}: no planned step, nudge if skipped", "planning") + # Gating decisions derived from the plan. # - Empty plan → fail-open: behave like before (memory + router). # - Plan with `searchMemory` directive → run memory enrichment. diff --git a/src/jarvis/reply/reply.spec.md b/src/jarvis/reply/reply.spec.md index b0109da..88c3f70 100644 --- a/src/jarvis/reply/reply.spec.md +++ b/src/jarvis/reply/reply.spec.md @@ -137,7 +137,8 @@ Design principles enforced by the engine: Compensates for small routers that misroute follow-ups where the user is supplying the missing info (field trace 2026-05-03: turn 1 invoked `getWeather` with no location configured, the tool returned `success=False`, the assistant relayed the request, turn 2 was "I'm in London", router picked `webSearch`, planner web-searched "weather in london tomorrow", Wikipedia fallback returned "Edge of Tomorrow" and the assistant parroted the film summary as the weather answer). A successful chain followed by a genuine new short ask ("log my breakfast") correctly does NOT carry over the prior tool — its `tool_failed=False` flag short-circuits the walker. The walker stops at the first genuine user message, walks both calling protocols (native: `assistant.tool_calls[*].function.name` matched to `role=tool` results by `tool_call_id`; text-tool fallback: `role=user` messages tagged with `tool_name`), and only collects names whose matching tool result message has `tool_failed=True`. The augmentation is an engine-side per-turn overlay: the router cache stores only the raw router output, so identical-query replays in future turns are unaffected. When carry-over fires, `_selection_source` becomes `+carryover` (or `+plan+carryover`) so the printed `🔧 Tools` log line stays honest. The flag distinguishes only success vs failure, not failure mode (argument issue vs network vs anything else); the user is most likely to follow up with a correction either way, and the chat model can still pick a different tool from the widened list. Edge cases: an MCP tool unloaded between turns is filtered out by the `_full_catalog_names` membership check (so a stale name never leaks into the schema). A tool turn evicted from `DialogueMemory._tool_turns` by the storage cap (`_tool_turns_max_storage`, default 16) loses its carry-over protection — acceptable because active sessions rarely accumulate 16 tool turns before reaching the recent-window boundary, and the chat model can still call `toolSearchTool` to re-widen mid-loop. Orphan assistant `tool_calls` (no matching `role=tool` result in the recent window — possible after truncation or scrub) are ignored and logged via `debug_log` so upstream data loss is diagnosable rather than silent. - - The per-turn allow-list exposed to the chat model is: `` + `` + `stop` (the sentinel) + `toolSearchTool`. + - **Waiting tools** (v2): a tool still holding something for the asker is unioned into the allow-list after the carry-over, since the message that answers it names no tool: `pcControl` while a request asked in the asker's previous reply waits for this message (`pc_control.awaits_answer`, the consent turn: "네." to "계산기를 실행해도 될까요?"), and `botBrowser` while the bot browser holds the page it opened for the asker (`bot_browser.has_page`, "소리 줄여 줘" while their video plays). Only tools in the catalogue; an engine-side per-turn overlay like the carry-over (the router cache is untouched); logged as `waiting tools kept on offer`. They are added after the intent judge's gates, so a turn the router gave no tool (thanks, small talk) still skips the planner. When the router did give the turn a tool, the planner planned no tool step and a tool holds something for the asker, the turn gets the one-step nudge for that tool (an answer without calling it is asked once to call it first): the planner cannot see what the tool holds, and the model said "소리를 절반으로 줄였습니다" with nothing done (eval `evals/test_bot_browser_tool.py`, volume case: 0/6 before the nudge, 8/8 after; before the volume action and the waiting tools 0/3, reading the person's own windows and saying it cannot). Live 2026-11-18 (0.1.40): the router left both out, so the consent call was refused ("no tool can start the calculator") and the volume request went to the person's own browser window. + - The per-turn allow-list exposed to the chat model is: `` + `` + `` + `stop` (the sentinel) + `toolSearchTool`. - `toolSearchTool` wraps the same routing logic (`select_tools`) but is invokable mid-loop. It takes a refined natural-language description of what the model is trying to accomplish and returns the expanded set of candidate tools. When invoked, the returned tools are merged into the allow-list for subsequent turns (still plus `stop` and `toolSearchTool` itself). This gives the agent a single-shot escape hatch when the initial routing was too narrow without widening the allow-list to "everything" by default. - `toolSearchTool` is a builtin; see `src/jarvis/tools/builtin/tool_search.spec.md`. diff --git a/src/jarvis/tools/builtin/bot_browser.py b/src/jarvis/tools/builtin/bot_browser.py index 368700e..c4ea82d 100644 --- a/src/jarvis/tools/builtin/bot_browser.py +++ b/src/jarvis/tools/builtin/bot_browser.py @@ -18,7 +18,7 @@ from ...debug import debug_log FENCE_START = "<<>>" FENCE_END = "<<>>" -_ACTIONS = ("open", "read", "click", "type", "back", "play") +_ACTIONS = ("open", "read", "click", "type", "back", "play", "volume") PAGE_WAIT_S = 15.0 # play: two pages (the results and the video) and the wait for the video to start. PLAY_WAIT_S = 25.0 @@ -39,6 +39,7 @@ const controls = () => [...document.querySelectorAll( const media = () => [...document.querySelectorAll('video,audio')].filter(m => m.currentSrc || m.src) .map(m => ({playing: !m.paused && !m.ended, time: m.currentTime, duration: isFinite(m.duration) ? m.duration : null, muted: m.muted || m.volume === 0, + volume: Math.round(m.volume * 100), main: m.tagName === 'VIDEO' && m.getBoundingClientRect().width >= innerWidth * 0.5, failed: !!m.error})); const sameTab = () => { @@ -68,6 +69,13 @@ const a = [...document.querySelectorAll('a[href*="/watch?v="]')].find(e => vis(e && !e.closest('ytd-ad-slot-renderer,ytd-promoted-video-renderer,ytd-in-feed-ad-layout-renderer,[is-ad]')); if (!a) return null; const u = new URL(a.getAttribute('href'), location.href); return u.origin + u.pathname + '?v=' + encodeURIComponent(u.searchParams.get('v')); })()""" +# The sound of the page set to a level (0 to 1): every sound and video, and a player that keeps its +# own volume (YouTube's sets it again on its next video) told too; above 0 it is unmuted. +_VOLUME = r""" +((l) => { const p = document.getElementById('movie_player'); + if (p && p.setVolume) { p.setVolume(Math.round(l * 100)); if (l > 0 && p.unMute) p.unMute(); } + const m = [...document.querySelectorAll('video,audio')].filter(x => x.currentSrc || x.src); + m.forEach(x => { x.volume = l; x.muted = l === 0; }); return m.length; })(%s)""" _CLICK = "(() => {" + _HELPERS + r""" return ((t) => { const m = /^\s*\[(\d+)\]\s*$/.exec(t); let el = null; if (m) el = (window.__jvItems || [])[Number(m[1]) - 1]; @@ -261,6 +269,12 @@ class BrowserSession: raise BrowserError("no video came up for those words") time.sleep(0.25) + def volume(self, level: int) -> Dict[str, Any]: + if not self._eval(_VOLUME % json.dumps(level / 100)): + raise BrowserError("no sound or video on this page") + time.sleep(min(0.2, max(0.0, self.deadline - time.monotonic()))) + return self.read() + def _press_play(self) -> None: """On a page just reached whose main player did not start: look again and press play.""" try: @@ -295,7 +309,7 @@ MAX_TABS = 5 _lock = threading.Lock() # Navigations that are safe to send again on a fresh connection; a click, typing or back may have # taken effect before the connection broke, so they are never repeated. -_REPEATABLE = ("open", "read", "play") +_REPEATABLE = ("open", "read", "play", "volume") def _action(raw: Any) -> str: @@ -348,7 +362,8 @@ def _media(page: Dict[str, Any]) -> str: d = m.get("duration") parts.append(("playing" if m.get("playing") else "paused") + " " + _clock(m.get("time")) + (f" of {_clock(d)}" if isinstance(d, (int, float)) and d > 0 else "") - + (" (muted)" if m.get("muted") else "")) + + (" (muted)" if m.get("muted") else + f", volume {m['volume']}%" if isinstance(m.get("volume"), int) else "")) return f"Media: {'; '.join(parts)}\n" if parts else "" @@ -378,10 +393,12 @@ class BotBrowserTool(Tool): @property def description(self) -> str: return ( - "Open and use a website in Jarvis's own web browser: search on a site, click, type into " - "a box, read the page. " + "Jarvis's own browser: action='open' a site, 'read', 'click', 'type', 'play' a video, " + "'volume' with level 0-100. " "Actions: open an address, read the current page, click a link or button by its label, " - "type into a search box or field (optionally pressing Enter), go back, or play. To play a " + "type into a search box or field (optionally pressing Enter), go back, play, or set the " + "volume of the sound or video on the page (the page shows its volume now, so for " + "\"half\" give half of that). To play a " "song or video, call play once with the words as query, exactly as the user said them and " "in their language (never translated): it searches YouTube, opens the first video and " "starts it, and the page that comes back says whether it truly plays. To search on a " @@ -400,7 +417,7 @@ class BotBrowserTool(Tool): "type": "object", "properties": { "action": {"type": "string", "enum": list(_ACTIONS), - "description": "open, read, click, type, back or play"}, + "description": "open, read, click, type, back, play or volume"}, "url": {"type": "string", "description": "for open: the web address"}, "query": {"type": "string", "description": "for open: words to search for on that site, as " "plain text (never percent-encoded); they go into the address's empty search " @@ -410,6 +427,7 @@ class BotBrowserTool(Tool): "text": {"type": "string", "description": "for type: what to type"}, "field": {"type": "string", "description": "for type: the field's label or placeholder (optional)"}, "submit": {"type": "boolean", "description": "for type: press Enter after typing (true to run a search)"}, + "level": {"type": "number", "description": "for volume: 0 (silent) to 100 (full)"}, }, "required": ["action"], } @@ -460,6 +478,12 @@ class BotBrowserTool(Tool): call = lambda s: s.type(text, args.get("field") or None, bool(args.get("submit"))) elif action == "back": call = lambda s: s.back() + elif action == "volume": + m = re.search(r"\d+(?:\.\d+)?", str(args.get("level") if args.get("level") is not None else "")) + if not m: + return _fail("volume needs a level from 0 to 100") + level = min(100, round(float(m.group()))) + call = lambda s: s.volume(level) else: call = lambda s: s.read() who = str(getattr(context, "user_id", "") or "") @@ -492,6 +516,13 @@ class BotBrowserTool(Tool): return ToolExecutionResult(success=True, reply_text=_render(page)) +def has_page(who: str) -> bool: + """Whether the bot browser holds a page it opened for this person (the reply engine keeps the + tool on offer then, reply.spec.md "Waiting tools"). Read without the lock: an action of + another person may hold it for its whole wait.""" + return bool(who) and who in _sessions + + def _keep(who: str, session: Any) -> None: """Keep a person's session as the most recent; close the oldest beyond MAX_TABS.""" _sessions[who] = session diff --git a/src/jarvis/tools/builtin/bot_browser.spec.md b/src/jarvis/tools/builtin/bot_browser.spec.md index 58ecd95..6aba78c 100644 --- a/src/jarvis/tools/builtin/bot_browser.spec.md +++ b/src/jarvis/tools/builtin/bot_browser.spec.md @@ -29,8 +29,12 @@ read it, click a link or button, type into a field, go back. Product rules: `doc - **One tab, also for "new tab" links.** Before a click or typing, links, forms and `` lose their `target` and `window.open` loads the address in the same tab, so what opens is what the tool reads next. -- **Never twice.** When the connection breaks during an action, `open`, `read` and `play` (which - only opens pages) are tried once more on a fresh connection (a fresh tab); `click`, `type` and `back` are not, because they may +- **Kept on offer while a page is open.** While the tool holds a page for the asker + (`has_page`), the reply engine offers it whatever the router picks (reply.spec.md, "Waiting + tools"): "소리 줄여 줘" or "화면에 오류 떴어" about the video it plays names no tool, and live + (2026-11-18) the model then worked on the person's own browser window through pcControl. +- **Never twice.** When the connection breaks during an action, `open`, `read`, `play` (which + only opens pages) and `volume` (which sets a level) are tried once more on a fresh connection (a fresh tab); `click`, `type` and `back` are not, because they may already have taken effect: the tool says it may or may not have happened. - **The fence holds.** `<<<` and `>>>` in anything read from a page become `‹‹‹` and `›››`, so page text cannot close or reopen the fence. @@ -47,11 +51,12 @@ One tool, `action` picks the operation (read as its first known word, so `open_p | `click` | `target` | clicks the link or button whose label is `target` (an exact label first, else the one containing it; `[n]` from the last list also works) | | `type` | `text` or `query` (`field`, `submit`) | types into the field whose label, placeholder or name contains `field` (else the focused field, else the first search or text box); `submit` presses Enter | | `back` | | goes back one page | +| `volume` | `level` | sets the sound of the page to `level` (0 to 100, the first number in it, above 100 is 100): every `