tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

feat(controlBrowser): add moveMouse (hover) action for the visible cursor

Browse Source 
The tool had no cursor-move/hover action, so "move the mouse to the search box"
had nothing to call and a weak model just claimed it had moved it. Add a
moveMouse action wired to human.humanHover (real xdotool cursor), targetable by
CSS selector or site= (that site's search box). Also clarify in the tool
description that 'navigate' types into the address bar (no mouse) while
'search'/'type' move the real cursor to the on-page box and click before
typing, so the model picks the visible-cursor path when the user wants it.
This commit is contained in:
javis-bot
2026-06-24 19:14:04 +09:00
parent 83999a5b0b
commit 5629da7e9f
3 changed files with 83 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
bot/scripts/stream-test/chrome-control.mjs
View File

@@ -17,6 +17,7 @@
// status | listTabs
// navigate {url} | back | forward | refresh
// newTab {url?} | closeTab {index?} | activateTab {index} | closePopups
// moveMouse {selector | site} (hover the real cursor, no click)
// click {selector} | type {text, selector?} | scroll {dir, notches?}
// pressKey {key} | screenshot {path}
import { chromium } from 'playwright';
@@ -40,6 +41,15 @@ if (!action) { out({ ok: false, error: 'no action' }); process.exit(1); }
const norm = (u) => (/^https?:\/\//i.test(u) ? u : `https://${u}`);
// Per-site homepage + search-box selector, shared by `search` and `moveMouse`.
const SITES = {
naver: { home: 'https://www.naver.com', box: '#query, input[name="query"]' },
google: { home: 'https://www.google.com', box: 'textarea[name="q"], input[name="q"]' },
daum: { home: 'https://www.daum.net', box: '#q, input[name="q"]' },
youtube: { home: 'https://www.youtube.com', box: 'input#search, input[name="search_query"]' },
bing: { home: 'https://www.bing.com', box: '#sb_form_q, input[name="q"]' },
};
// The genuinely-active tab is the one whose document is visible. Playwright has
// no "active page" accessor over CDP, so probe visibilityState (fixes treating
// tab 0 as active and breaking sequential ops on a specific tab).
@@ -103,13 +113,6 @@ try {
const q = String(cmd.query || '').trim();
if (!q) throw new Error('search: no query');
const siteKey = String(cmd.site || 'google').toLowerCase();
const SITES = {
naver: { home: 'https://www.naver.com', box: '#query, input[name="query"]' },
google: { home: 'https://www.google.com', box: 'textarea[name="q"], input[name="q"]' },
daum: { home: 'https://www.daum.net', box: '#q, input[name="q"]' },
youtube: { home: 'https://www.youtube.com', box: 'input#search, input[name="search_query"]' },
bing: { home: 'https://www.bing.com', box: '#sb_form_q, input[name="q"]' },
};
const s = SITES[siteKey] || SITES.google;
await front(page);
// 1) Go to the homepage.
@@ -205,6 +208,26 @@ try {
break;
}
case 'moveMouse': {
// Move/hover the REAL cursor onto an element WITHOUT clicking. Target is a
// CSS selector, or site=naver/google/... for that site's search box.
// Only meaningful with xdotool (the visible cursor); with no xdotool there
// is no cursor to move, so report that rather than faking success.
const siteKey = String(cmd.site || '').toLowerCase();
const selector = String(cmd.selector || '').trim() || (SITES[siteKey] ? SITES[siteKey].box : '');
if (!selector) throw new Error('moveMouse: no selector or known site');
if (!(HAS_XDOTOOL && cmd.human !== false)) {
out({ ok: false, error: 'no xdotool: cannot move the visible cursor on this host' });
break;
}
await front(page);
const locator = page.locator(selector).first();
await locator.waitFor({ state: 'visible', timeout: 10000 }).catch(() => {});
await human.humanHover(page, locator);
out({ ok: true, target: cmd.selector || siteKey, input: 'human' });
break;
}
case 'click': {
const selector = String(cmd.selector || '').trim();
if (!selector) throw new Error('click: no selector');