tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

feat(controlBrowser): add moveMouse (hover) action for the visible cursor

Browse Source 
The tool had no cursor-move/hover action, so "move the mouse to the search box"
had nothing to call and a weak model just claimed it had moved it. Add a
moveMouse action wired to human.humanHover (real xdotool cursor), targetable by
CSS selector or site= (that site's search box). Also clarify in the tool
description that 'navigate' types into the address bar (no mouse) while
'search'/'type' move the real cursor to the on-page box and click before
typing, so the model picks the visible-cursor path when the user wants it.
This commit is contained in:
javis-bot
2026-06-24 19:14:04 +09:00
parent 83999a5b0b
commit 5629da7e9f
3 changed files with 83 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
src/jarvis/tools/builtin/control_browser.py
View File

@@ -45,9 +45,16 @@ class ControlBrowserTool(Tool):
"Use this (NOT webSearch) whenever the user wants something done or shown IN the "
"browser on screen: open a website or URL, search on a specific site (action "
"'search' with site=naver/google/daum/youtube/bing), go back/forward, refresh, "
"manage tabs (list/new/close/switch), close popups, click, type, scroll, or "
"screenshot. webSearch only returns text and shows nothing on screen; this tool "
"actually navigates the visible browser. Only available in screen-share mode. "
"manage tabs (list/new/close/switch), close popups, move the mouse onto an element, "
"click, type, scroll, or screenshot. webSearch only returns text and shows nothing "
"on screen; this tool actually navigates the visible browser. "
"Cursor behaviour matters: 'search' and 'type' (with a selector) move the REAL mouse "
"cursor to the on-page box, click it, then type one character at a time — use these "
"when the user wants to search or type on a page. 'navigate' only types the URL into "
"the address bar (no mouse movement). 'moveMouse' moves/hovers the visible cursor "
"onto an element (selector, or site=naver/... for that site's search box) WITHOUT "
"clicking — use it when the user just asks to move the mouse somewhere. "
"Only available in screen-share mode. "
"Never claim you did any of this unless this tool returns success."
)
@@ -61,16 +68,16 @@ class ControlBrowserTool(Tool):
"enum": [
"status", "listTabs", "navigate", "search", "back",
"forward", "refresh", "newTab", "closeTab", "activateTab",
"closePopups", "click", "type", "scroll", "pressKey",
"closePopups", "moveMouse", "click", "type", "scroll", "pressKey",
"screenshot",
],
"description": "What to do in the browser.",
},
"url": {"type": "string", "description": "Target URL/site for navigate/newTab (e.g. 'naver.com')."},
"query": {"type": "string", "description": "Search text for action 'search'."},
"site": {"type": "string", "description": "Search site for action 'search': naver, google, daum, youtube, bing."},
"site": {"type": "string", "description": "Site for action 'search' or 'moveMouse': naver, google, daum, youtube, bing. For moveMouse it targets that site's search box."},
"index": {"type": "integer", "description": "Tab index for closeTab/activateTab (from listTabs)."},
"selector": {"type": "string", "description": "CSS selector for click/type."},
"selector": {"type": "string", "description": "CSS selector for click/type/moveMouse."},
"text": {"type": "string", "description": "Text to type."},
"key": {"type": "string", "description": "Key to press, e.g. 'Return', 'Escape'."},
"dir": {"type": "string", "description": "Scroll direction: 'down' or 'up'."},
@@ -164,6 +171,9 @@ class ControlBrowserTool(Tool):
return f"탭 {data.get('active')}번으로 전환했습니다."
if action == "closePopups":
return f"팝업/빈 탭 {data.get('closed')}개를 닫았습니다."
if action == "moveMouse":
target = data.get("target") or args.get("selector") or args.get("site") or "대상"
return f"마우스 커서를 {target} 위치로 옮겼습니다."
if action == "screenshot":
return f"화면을 캡처했습니다: {data.get('path')}"
return "완료했습니다."