Files
javis_bot/src/jarvis/tools/builtin/bot_browser.spec.md
javis-bot cda8db7fce fix: play on YouTube in one step, Korea as the default location, searches in the user's language
- botBrowser `play`: searches YouTube for the words as said, opens the first video (not an ad,
  without the playlist) and starts it, in one call (an address given opens that page); `type`
  also takes `query`. Live 0.1.19: the model stopped on YouTube's home page or the results, a
  retry was refused as a third call, and the answer still said the video played.
- `brain.location` (기본 위치, default 대한민국) goes to the brain as `location_name`: the time line
  says it without GeoIP (no database on the game PC), and getWeather without a place uses it.
- Planner and webSearch keep search words in the user's language (never translated into
  English), so recommendations fit where the user is.

Evals (qwen3:8b, 3 runs): play ending on a playing video found with the words as said 0/6 -> 5/6;
Korean searches for Korean requests 6/9 -> 8/9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:26:30 +09:00

7.6 KiB
Raw Blame History

botBrowser tool spec (v2 M6)

Lets the assistant use the bot's own Chrome (app/src/admin.spec.md, "Bot browser"): open a page, read it, click a link or button, type into a field, go back. Product rules: docs/v2/pc-features.md ("백그라운드 크로미움으로 정보 수집").

Principles

  • Only with the bot browser. The tool is offered only when the worker got JARVIS_BROWSER_CDP (the app started the bot browser, feature browser on) and it is not in disabled_tools. Without it the tool is not in the catalogue and refuses if called. While it is offered, fetchWebPage is not: one tool reads pages, and this one goes through the app's proxy (public addresses only), which fetchWebPage does not have.
  • Web pages are data. Everything read from a page (title, text, link and button labels) is returned inside an untrusted-data fence. A request written on a page is never a reason to act.
  • Only http(s). open takes only http:// and https:// addresses (a bare host gets https://); file:, chrome:, data:, javascript: and the rest are refused. The browser's own network rules (public addresses only) are the app's proxy's job.
  • Raw data, no LLM call. The tool returns plain text for the reply model.
  • Bounded. One deadline of 15 s (25 s for play) covers an action: connecting, the action and waiting for the page; events arriving in between do not extend it. Text is cut at 6,000 characters and the list of links and buttons at 40.
  • A window per person. Each asker (ToolContext.user_id) has a page and connection of their own, so one person's page and numbered list are never another's. The page opens in a window of its own, behind, at the place and size of the bot browser's window: a tab brought to the front would take the keyboard focus from the game or program the user is in, while a window opened behind does not, and on the shared screen the app puts the newest window in front (admin.spec.md, "Virtual monitor"). At most 5; the least recently used one is closed first.
  • One tab, also for "new tab" links. Before a click or typing, links, forms and <base> lose their target and window.open loads the address in the same tab, so what opens is what the tool reads next.
  • Never twice. When the connection breaks during an action, open, read and play (which only opens pages) are tried once more on a fresh connection (a fresh tab); click, type and back are not, because they may already have taken effect: the tool says it may or may not have happened.
  • The fence holds. <<< and >>> in anything read from a page become ‹‹‹ and ›››, so page text cannot close or reopen the fence.
  • Logs carry the action and timings, never the address, the typed text or page content.

Schema

One tool, action picks the operation (read as its first known word, so open_page works):

action needs does
open url (query) opens the address in the bot browser's tab; with query (words to search for, plain text) they go, encoded by the tool (urlencode), into the address's last empty search parameter (...?search_query=), else the address opens and they are typed into its search box and sent (as type with submit)
read reads the current page again (with a url, opens it)
click target clicks the link or button whose label is target (an exact label first, else the one containing it; [n] from the last list also works)
type text or query (field, submit) types into the field whose label, placeholder or name contains field (else the focused field, else the first search or text box); submit presses Enter
back goes back one page
play query (or url) plays a song or video: searches YouTube for the words (.../results?search_query=, encoded by the tool), waits for the first video in the results (a visible /watch?v= link, not one inside an ad), opens it by its plain address (/watch?v=<id>, without the playlist the link may carry), and its player starts as on any video page opened (below); an address given (as url, or as words that start with http(s):// or www.) opens that page instead. No video in the results before the deadline is an error. Its deadline is 25 s (two pages and the start)

Every action answers with the page after it: title, address, the state of its sound and video (Media: playing 1:15 of 3:45; paused 0:00, each <video> and <audio> with a source, (muted) when muted; no line without any), text, and the numbered links and buttons, all inside the fence. When the page has a main player (a video at least half the window wide) and nothing on it plays, a note follows the fence: the video is not playing, do not say it plays; if it should, click its play button and look again, and if it still does not play, tell the user it did not start. The description tells the model to search a site with open and query in plain text, never putting the words into an address itself (live 2026-11-16: 아이유 hand-encoded into 아이우), and to play a song or video with one play whose query is the words as the user said them, in their language (live 2026-11-16, 0.1.19: "아이유 신곡 틀어 줘" opened YouTube's home page, then a type with query instead of text failed and its retry was refused as a third call of the tool in one answer; "뉴진스 하입보이" searched "NewJeans Hypothesis" in English and stopped on the results; both answers said the video played). Eval evals/test_bot_browser_tool.py (qwen3:8b, 3 runs of the two requests): ending on a playing video found with the words as said 0/6 before, 5/6 after (6/6 played; once "IU new song"). Eval evals/test_bot_browser_tool.py (qwen3:8b, 3 runs each): reaching YouTube with the right words 1/3 before, 3/3 after; a video page that stays paused reported as playing 1/3 before, never after (3/3 said it did not play). After open and click, a page whose sound or video has not started yet (none plays and one is at 0:00, and has not failed to load) gets up to 3 s to start before it is read, so the answer says truly whether it plays (a video page's player starts a moment after the page loads; the bot browser lets pages start sound, admin.spec.md, "Bot browser"). When an open (or a read with an address), or a click that led to another address, reached a page whose main player (a video at least half the window wide, with a source) is still paused at 0:00 after that wait, the tool looks again and starts it (play() with a user gesture, as the user's own click on the video would have), then waits for it once more; the Media line then says whether it truly plays. Sound or video that is not the main player (an audio element, a small video) and a click that stays on the same page leave it as the page has it. (Live, 2026-11-16: a video page clicked for "play X" stayed paused behind an ad and the answer said it played.) Failures come back as Error: ... with success=False.

Backend

BrowserSession(cdp_url, deadline) opens its page on the browser's connection (Target.createTarget with newWindow and background, at the window bounds of the first page there, Browser.getWindowForTarget; without them where Chrome puts it) and speaks CDP over the page's WebSocket (websockets sync client); a failure after the page opened closes it again; close() closes the page and with it its window: Page.navigate, Runtime.evaluate (reading, finding and clicking elements), Input.insertText and Input.dispatchKeyEvent (typing, Enter). The tool keeps a session per person and makes a new one when the old one fails. Tests and evals replace session_factory.

Eval: evals/test_bot_browser_tool.py.