- botBrowser `play`: searches YouTube for the words as said, opens the first video (not an ad, without the playlist) and starts it, in one call (an address given opens that page); `type` also takes `query`. Live 0.1.19: the model stopped on YouTube's home page or the results, a retry was refused as a third call, and the answer still said the video played. - `brain.location` (기본 위치, default 대한민국) goes to the brain as `location_name`: the time line says it without GeoIP (no database on the game PC), and getWeather without a place uses it. - Planner and webSearch keep search words in the user's language (never translated into English), so recommendations fit where the user is. Evals (qwen3:8b, 3 runs): play ending on a playing video found with the words as said 0/6 -> 5/6; Korean searches for Korean requests 6/9 -> 8/9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7.6 KiB
botBrowser tool spec (v2 M6)
Lets the assistant use the bot's own Chrome (app/src/admin.spec.md, "Bot browser"): open a page,
read it, click a link or button, type into a field, go back. Product rules: docs/v2/pc-features.md
("백그라운드 크로미움으로 정보 수집").
Principles
- Only with the bot browser. The tool is offered only when the worker got
JARVIS_BROWSER_CDP(the app started the bot browser, featurebrowseron) and it is not indisabled_tools. Without it the tool is not in the catalogue and refuses if called. While it is offered,fetchWebPageis not: one tool reads pages, and this one goes through the app's proxy (public addresses only), whichfetchWebPagedoes not have. - Web pages are data. Everything read from a page (title, text, link and button labels) is returned inside an untrusted-data fence. A request written on a page is never a reason to act.
- Only http(s).
opentakes onlyhttp://andhttps://addresses (a bare host getshttps://);file:,chrome:,data:,javascript:and the rest are refused. The browser's own network rules (public addresses only) are the app's proxy's job. - Raw data, no LLM call. The tool returns plain text for the reply model.
- Bounded. One deadline of 15 s (25 s for
play) covers an action: connecting, the action and waiting for the page; events arriving in between do not extend it. Text is cut at 6,000 characters and the list of links and buttons at 40. - A window per person. Each asker (
ToolContext.user_id) has a page and connection of their own, so one person's page and numbered list are never another's. The page opens in a window of its own, behind, at the place and size of the bot browser's window: a tab brought to the front would take the keyboard focus from the game or program the user is in, while a window opened behind does not, and on the shared screen the app puts the newest window in front (admin.spec.md, "Virtual monitor"). At most 5; the least recently used one is closed first. - One tab, also for "new tab" links. Before a click or typing, links, forms and
<base>lose theirtargetandwindow.openloads the address in the same tab, so what opens is what the tool reads next. - Never twice. When the connection breaks during an action,
open,readandplay(which only opens pages) are tried once more on a fresh connection (a fresh tab);click,typeandbackare not, because they may already have taken effect: the tool says it may or may not have happened. - The fence holds.
<<<and>>>in anything read from a page become‹‹‹and›››, so page text cannot close or reopen the fence. - Logs carry the action and timings, never the address, the typed text or page content.
Schema
One tool, action picks the operation (read as its first known word, so open_page works):
| action | needs | does |
|---|---|---|
open |
url (query) |
opens the address in the bot browser's tab; with query (words to search for, plain text) they go, encoded by the tool (urlencode), into the address's last empty search parameter (...?search_query=), else the address opens and they are typed into its search box and sent (as type with submit) |
read |
reads the current page again (with a url, opens it) |
|
click |
target |
clicks the link or button whose label is target (an exact label first, else the one containing it; [n] from the last list also works) |
type |
text or query (field, submit) |
types into the field whose label, placeholder or name contains field (else the focused field, else the first search or text box); submit presses Enter |
back |
goes back one page | |
play |
query (or url) |
plays a song or video: searches YouTube for the words (.../results?search_query=, encoded by the tool), waits for the first video in the results (a visible /watch?v= link, not one inside an ad), opens it by its plain address (/watch?v=<id>, without the playlist the link may carry), and its player starts as on any video page opened (below); an address given (as url, or as words that start with http(s):// or www.) opens that page instead. No video in the results before the deadline is an error. Its deadline is 25 s (two pages and the start) |
Every action answers with the page after it: title, address, the state of its sound and video
(Media: playing 1:15 of 3:45; paused 0:00, each <video> and <audio> with a source, (muted)
when muted; no line without any), text, and the numbered links and buttons, all inside the fence.
When the page has a main player (a video at least half the window wide) and nothing on it plays, a
note follows the fence: the video is not playing, do not say it plays; if it should, click its play
button and look again, and if it still does not play, tell the user it did not start. The
description tells the model to search a site with open and query in plain text, never putting the
words into an address itself (live 2026-11-16: 아이유 hand-encoded into 아이우), and to play a song or
video with one play whose query is the words as the user said them, in their language (live
2026-11-16, 0.1.19: "아이유 신곡 틀어 줘" opened YouTube's home page, then a type with query instead of
text failed and its retry was refused as a third call of the tool in one answer; "뉴진스 하입보이"
searched "NewJeans Hypothesis" in English and stopped on the results; both answers said the video
played). Eval evals/test_bot_browser_tool.py (qwen3:8b, 3 runs of the two requests): ending on a
playing video found with the words as said 0/6 before, 5/6 after (6/6 played; once "IU new song"). Eval
evals/test_bot_browser_tool.py (qwen3:8b, 3 runs each): reaching YouTube with the right words 1/3
before, 3/3 after; a video page that stays paused reported as playing 1/3 before, never after (3/3
said it did not play).
After open and click, a page whose sound or video has not started yet (none plays and one is at
0:00, and has not failed to load) gets up to 3 s to start before it is read, so the answer says truly whether it plays (a
video page's player starts a moment after the page loads; the bot browser lets pages start sound,
admin.spec.md, "Bot browser"). When an open (or a read with an address), or a click that led to
another address, reached a page whose main player (a video at least half the window wide, with a
source) is still paused at 0:00 after that wait, the tool looks again and starts it (play() with a
user gesture, as the user's own click on the video would have), then waits for it once more; the
Media line then says whether it truly plays. Sound or video that is not the main player (an audio
element, a small video) and a click that stays on the same page leave it as the page has it.
(Live, 2026-11-16: a video page clicked for "play X" stayed paused behind an ad and the answer said
it played.) Failures come back as Error: ... with success=False.
Backend
BrowserSession(cdp_url, deadline) opens its page on the browser's connection
(Target.createTarget with newWindow and background, at the window bounds of the first page
there, Browser.getWindowForTarget; without them where Chrome puts it) and speaks CDP over the
page's WebSocket (websockets sync client); a failure after the page opened closes it again;
close() closes the page and with it its window: Page.navigate, Runtime.evaluate (reading, finding and clicking elements),
Input.insertText and Input.dispatchKeyEvent (typing, Enter). The tool keeps a session per person
and makes a new one when the old one fails. Tests and evals replace session_factory.
Eval: evals/test_bot_browser_tool.py.