Compare commits
5 Commits
app-v0.1.4
...
app-v0.1.4
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3af881935c | ||
|
|
20bd90f65e | ||
|
|
9a296a5683 | ||
|
|
5767ffa5ae | ||
|
|
378ccfafad |
2
app/Cargo.lock
generated
2
app/Cargo.lock
generated
@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.42"
|
||||
version = "0.1.44"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.42"
|
||||
version = "0.1.44"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"
|
||||
|
||||
4
app/shell/Cargo.lock
generated
4
app/shell/Cargo.lock
generated
@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.42"
|
||||
version = "0.1.44"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
@@ -2666,7 +2666,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.42"
|
||||
version = "0.1.44"
|
||||
dependencies = [
|
||||
"jarvis-app",
|
||||
"serde_json",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.42"
|
||||
version = "0.1.44"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"
|
||||
|
||||
@@ -230,9 +230,7 @@ agent-desktop and decides who may.
|
||||
private to the person: agent-desktop refuses a state folder others may write to, which a folder
|
||||
outside the user's profile is by inheritance), a 15 s limit per command. A
|
||||
program it starts gets the person's own APPDATA and LOCALAPPDATA back and starts in their user
|
||||
folder. Actions are headless (no cursor, no focus), except `keys`, which brings the window to the
|
||||
front first (`focus-window`, then `press`); when Windows does not let it come forward (its
|
||||
foreground lock), no key is sent ("그 창을 앞으로 가져오지 못해 키를 보내지 않았습니다"). `--force`
|
||||
folder. Actions are headless (no cursor, no focus), `keys` as below. `--force`
|
||||
(which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc and the like) and the raw mouse and key
|
||||
commands are never used; what the brain sends is checked so nothing in it reads as an option,
|
||||
and typed text goes in through stdin (`set-value --stdin`), never in the arguments. The commands: `list-windows`, `snapshot
|
||||
@@ -243,6 +241,38 @@ agent-desktop and decides who may.
|
||||
떠 있을 수 있습니다). 창을 다시 읽어 주세요", and with several of it running close-app cannot
|
||||
choose: "그 프로그램이 여러 개 실행 중이라 이 창만 닫을 수 없습니다. 창의 닫기 단추를 눌러 주세요"),
|
||||
`restore`.
|
||||
- **Keys** (`keys`; user decision 2026-11-19): one way is chosen before any key is sent, from the
|
||||
keys and the window, and never tried the other way after (a key is never pressed twice):
|
||||
- **Behind** (the default): one key without Ctrl, Alt, Shift or Win that types or moves (`a`-`z`,
|
||||
`0`-`9`, `enter`, `tab`, `space`, `escape`, `delete`/`backspace`, `forwarddelete`, the arrows,
|
||||
`home`, `end`, `pageup`, `pagedown`, `f1`-`f9`, `f11`, `f12`; agent-desktop's names) is posted to
|
||||
the window's text control (`PostMessage`, key down then key up; the program's own message
|
||||
loop makes the character) without bringing the window forward: the person's screen and focus
|
||||
stay as they are. Only for a window with exactly one text control shown (`Edit`, `RichEdit20A`,
|
||||
`RichEdit20W`, `RICHEDIT50W`, the new Notepad's `RichEditD2DPT`, `Scintilla`), which gets the
|
||||
keys: Windows does not tell which control of a window behind has the focus (a thread behind
|
||||
has none; win-ci VM 2026-11-19), so keys go behind only where there is no doubt where they land.
|
||||
Store and WinUI apps (the new Calculator), Chromium, Electron and Firefox windows have none.
|
||||
- **In front for a moment**: a combination, any other key, and every key for a window that does
|
||||
not take keys behind or is minimised. A minimised window is restored inside the moment
|
||||
(restoring brings it forward, so the window in front is noted before and brought back after;
|
||||
the window itself stays restored). The window is brought to the front (`focus-window`, then `press`) and the
|
||||
window that was in front before is brought back right after (`focus-window` on it, also when
|
||||
`press` failed; one that cannot be brought back is left, with a warning in the log). For a
|
||||
window whose process agent-desktop cannot find from the window (the new Calculator, a Store
|
||||
app: `press --window-id` is `APP_NOT_FOUND` and sends nothing; game PC 2026-11-19) the keys
|
||||
are pressed without naming the window (nothing was sent before), into the window in front,
|
||||
only while that is still the window (checked just before). A key the
|
||||
person types in that moment may go to the window. When Windows does not let the window come
|
||||
forward (its foreground lock), no key is sent ("그 창을 앞으로 가져오지 못해 키를 보내지
|
||||
않았습니다"). A window already in front gets the keys there and nothing else moves.
|
||||
- **Never in front during a game or full screen**: while game mode is on, the window in front
|
||||
covers its whole monitor without a title bar or sizing border (a video or a program full
|
||||
screen; not the desktop), or Windows reports a full-screen Direct3D program, keys that need
|
||||
the window in front are not sent ("게임 중이거나 전체 화면이라 그 창을 앞으로 가져오지 않았습니다.
|
||||
키를 보내지 않았습니다").
|
||||
- Either way the window is read again after ("Results"); keys a program did not take behind show
|
||||
there as no change, and the brain does not say they were pressed.
|
||||
- **Windows**: as `list-windows` gives them, without this app's own windows, windows without a
|
||||
title and the programs of `pcControl.screenExclude` (never listed, read or acted on, as for
|
||||
screen viewing); at most 30. `window` is a window id or part of a title or program name (the
|
||||
@@ -287,15 +317,16 @@ agent-desktop and decides who may.
|
||||
interactive elements read afresh after the action (at most 200), so the brain says it was done
|
||||
only from what it sees; an action that closes its window (a dialog's button, Enter) answers the
|
||||
window with no elements. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
|
||||
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
|
||||
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting; for `keys` inside
|
||||
the moment in front, "Keys").
|
||||
- **Refused before asking**: the stop, the anti-cheat, an administrator window (a process whose
|
||||
elevation cannot be read counts as one) and a password field are refused without asking.
|
||||
`windows` and `read` at `ask` answer `ask: true` like the rest.
|
||||
- **Activity**: each action but `windows` and `read` is `pc.<action>` with the program's file name
|
||||
(never a title, element name or text), who asked and whether it worked, with the refusal's reason
|
||||
("Activity"); an `ask` answer is not recorded (nothing was done).
|
||||
- Logs carry the action, the result code and the time taken; never a title, element name, value or
|
||||
typed text.
|
||||
- Logs carry the action, the result code and the time taken (for `keys` also the way: behind or in
|
||||
front); never a title, element name, value, typed text or key.
|
||||
|
||||
## Personal notes (`app/src/notes.rs`)
|
||||
|
||||
@@ -1087,7 +1118,12 @@ a path with non-ASCII letters) or from a Battle.net game's install folder (a tes
|
||||
turns game mode on with its name and off once it ends. Screen viewing (`tests/screen_windows.rs`,
|
||||
`JARVIS_TEST_SCREEN=1`): the monitors and Notepad's window are listed, the window, each monitor and
|
||||
the whole screen come back as JPEG pictures of their size, and with Notepad excluded it is neither
|
||||
picked nor listed and its area is black on the whole screen; someone not registered is refused. On
|
||||
picked nor listed and its area is black on the whole screen; someone not registered is refused.
|
||||
Keys (`tests/pc_keys_windows.rs`, `JARVIS_TEST_PC_KEYS=1`, agent-desktop at `JARVIS_AGENT_DESKTOP`;
|
||||
it opens Notepad, Character Map and the Calculator, and refuses to start while any of them runs):
|
||||
with Character Map in front, letters and Enter reach Notepad behind it and Character Map stays in
|
||||
front; a combination and the Calculator's keys go in front for a moment and Character Map comes
|
||||
back. On
|
||||
every system with a fake screen: refusals before any capture, monitors by number, the whole screen
|
||||
as placed with excluded areas black (also at another display scale), an excluded window that
|
||||
moves or is brought back during the capture never shown (taken again, or refused while it keeps
|
||||
@@ -1100,7 +1136,9 @@ anti-cheat refusing every action but reading (its own processes and the person's
|
||||
game detection), the emergency stop refusing and lifted, and the real runner killing a running
|
||||
command when it is pressed and at its time limit, a password field and an administrator window
|
||||
refused before acting, readings capped at 200 elements (interactive first) and cut to 120
|
||||
characters, `STALE_REF`'s reason, a minimised window restored and `keys` brought to the front,
|
||||
characters, `STALE_REF`'s reason, a minimised window restored, `keys` sent behind or in front by the keys and the window (one way
|
||||
only, the window in front brought back, also after a failed press) and never in front during a
|
||||
game or full screen,
|
||||
excluded and own windows never listed, `run` reading its new window, the activity naming the
|
||||
program only, the hotkey and shortcut formats, and one pin of agent-desktop in `pc.rs`,
|
||||
`package.sh` and M0. Performance stages (on every system, with a fake
|
||||
|
||||
404
app/src/pc.rs
404
app/src/pc.rs
@@ -52,6 +52,7 @@ pub const STOPPED: &str = "비상 정지 중입니다. 단축키를 한 번 더
|
||||
pub const STALE: &str = "화면이 바뀌었습니다. 다시 읽어 주세요";
|
||||
const NO_WINDOW: &str = "그 창을 찾지 못했습니다";
|
||||
const NOT_FRONT: &str = "그 창을 앞으로 가져오지 못해 키를 보내지 않았습니다";
|
||||
const NOT_IN_FRONT_NOW: &str = "게임 중이거나 전체 화면이라 그 창을 앞으로 가져오지 않았습니다. 키를 보내지 않았습니다";
|
||||
const NOT_CLOSED: &str = "프로그램이 아직 닫히지 않았습니다(저장할지 묻는 창이 떠 있을 수 있습니다). 창을 다시 읽어 주세요";
|
||||
const CLOSE_SEVERAL: &str = "그 프로그램이 여러 개 실행 중이라 이 창만 닫을 수 없습니다. 창의 닫기 단추를 눌러 주세요";
|
||||
const RUNNING_SEVERAL: &str = "그 프로그램이 이미 여러 개 실행 중이라 새로 열지 못했습니다. 열린 창 중에서 골라 주세요";
|
||||
@@ -212,6 +213,24 @@ pub trait Desktop: Send + Sync {
|
||||
fn program(&self, name: &str) -> String {
|
||||
name.to_owned()
|
||||
}
|
||||
/// The window in front now, as agent-desktop names it (`w-<handle>`).
|
||||
fn foreground(&self) -> Option<String> {
|
||||
None
|
||||
}
|
||||
/// Whether the window in front fills its monitor without a frame (not the desktop), or a
|
||||
/// full-screen Direct3D program runs.
|
||||
fn fullscreen(&self) -> bool {
|
||||
false
|
||||
}
|
||||
/// Whether window `id` takes keys posted to it behind: it has one text control shown
|
||||
/// (`text_control`).
|
||||
fn takes_keys_behind(&self, _id: &str) -> bool {
|
||||
false
|
||||
}
|
||||
/// Post one key (`behind_key`) to window `id`'s text control, without bringing it forward.
|
||||
fn post_key(&self, _id: &str, _key: &str) -> Result<(), String> {
|
||||
Err("PLATFORM_NOT_SUPPORTED".into())
|
||||
}
|
||||
}
|
||||
|
||||
/// The real agent-desktop: one process per command, its APPDATA, LOCALAPPDATA and state folder in
|
||||
@@ -340,6 +359,28 @@ impl Desktop for Cli {
|
||||
#[cfg(not(windows))]
|
||||
name.to_owned()
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn foreground(&self) -> Option<String> {
|
||||
win::foreground()
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn fullscreen(&self) -> bool {
|
||||
win::fullscreen()
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn takes_keys_behind(&self, id: &str) -> bool {
|
||||
win::text_control_of(id).is_some()
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn post_key(&self, id: &str, key: &str) -> Result<(), String> {
|
||||
let vk = behind_key(key).ok_or_else(|| "INVALID_ARGS".to_owned())?;
|
||||
let to = win::text_control_of(id).ok_or_else(|| "WINDOW_NOT_FOUND".to_owned())?;
|
||||
win::post_key(to, vk)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the actions --------------------------------------------------------------------------------
|
||||
@@ -465,6 +506,42 @@ fn valid_keys(k: Option<&str>) -> Result<String, Fail> {
|
||||
if ok { Ok(k) } else { Err(refused("키를 알아듣지 못했습니다 (예: ctrl+s, enter)")) }
|
||||
}
|
||||
|
||||
/// The virtual key of `keys` when it can be sent behind: one key without modifiers that types or
|
||||
/// moves, by agent-desktop's names (`a`-`z` and `0`-`9` as their keys on a US layout); else `None`.
|
||||
fn behind_key(keys: &str) -> Option<u16> {
|
||||
let k = keys.to_ascii_lowercase();
|
||||
let vk = match k.as_str() {
|
||||
"return" | "enter" => 0x0D,
|
||||
"escape" | "esc" => 0x1B,
|
||||
"tab" => 0x09,
|
||||
"space" => 0x20,
|
||||
"delete" | "backspace" => 0x08,
|
||||
"forwarddelete" => 0x2E,
|
||||
"home" => 0x24,
|
||||
"end" => 0x23,
|
||||
"pageup" => 0x21,
|
||||
"pagedown" => 0x22,
|
||||
"left" => 0x25,
|
||||
"up" => 0x26,
|
||||
"right" => 0x27,
|
||||
"down" => 0x28,
|
||||
// F10 is Windows' menu key (a system key), so it goes in front.
|
||||
f if f.len() <= 3 && f.starts_with('f') => match f[1..].parse::<u16>() {
|
||||
Ok(n @ (1..=9 | 11 | 12)) => 0x6F + n,
|
||||
_ => return None,
|
||||
},
|
||||
c if c.len() == 1 && c.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit()) => c.as_bytes()[0].to_ascii_uppercase() as u16,
|
||||
_ => return None,
|
||||
};
|
||||
Some(vk)
|
||||
}
|
||||
|
||||
/// Whether a control of this class takes keys posted behind: the text controls whose keys Windows
|
||||
/// gives to the control itself (classic Edit and RichEdit, the new Notepad's, Scintilla).
|
||||
pub fn text_control(class: &str) -> bool {
|
||||
["Edit", "RichEdit20A", "RichEdit20W", "RICHEDIT50W", "RichEditD2DPT", "Scintilla"].contains(&class)
|
||||
}
|
||||
|
||||
/// A program to start by name: no folders, nothing that could be read as an option.
|
||||
fn valid_program(p: Option<&str>) -> Result<String, Fail> {
|
||||
let p = p.map(str::trim).filter(|p| !p.is_empty()).ok_or_else(|| refused("실행할 프로그램 이름이 필요합니다"))?;
|
||||
@@ -642,7 +719,8 @@ impl Pc {
|
||||
if self.stop.on() {
|
||||
return Err(refused(STOPPED));
|
||||
}
|
||||
if w.minimised {
|
||||
// Keys restore in their moment in front (restoring brings the window forward).
|
||||
if w.minimised && a.action != "keys" {
|
||||
self.cmd(&args(&["restore", "--window-id", &w.id]), None, &a.action)?;
|
||||
}
|
||||
match a.action.as_str() {
|
||||
@@ -652,11 +730,7 @@ impl Pc {
|
||||
"type" => {
|
||||
self.cmd(&args(&["set-value", element.unwrap(), "--stdin"]), Some(text), "type")?;
|
||||
}
|
||||
"keys" => {
|
||||
// Keys go to the window in front, so it is brought there first.
|
||||
self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys")?;
|
||||
self.cmd(&args(&["press", keys.as_deref().unwrap(), "--window-id", &w.id]), None, "keys")?;
|
||||
}
|
||||
"keys" => self.keys(&w, keys.as_deref().unwrap())?,
|
||||
_ => {
|
||||
self.cmd(&args(&["close-app", &w.program]), None, "close")?;
|
||||
return Ok(Done::Item(json!({"window": w.view(), "elements": []})));
|
||||
@@ -669,6 +743,50 @@ impl Pc {
|
||||
}
|
||||
}
|
||||
|
||||
/// Press `keys` in window `w` one way, chosen first (admin.spec.md, "Keys"): behind it when it is
|
||||
/// one plain key and the window takes it so, else in front for a moment (a minimised window
|
||||
/// restored in it) and the window that was in front brought back. Never both ways: a key is
|
||||
/// never pressed twice.
|
||||
fn keys(&self, w: &Win, keys: &str) -> Result<(), Fail> {
|
||||
let front = self.desktop.foreground();
|
||||
if front.as_deref() == Some(w.id.as_str()) {
|
||||
crate::debug_log!("conversation", "pc keys way=front-already");
|
||||
return self.press(w, keys);
|
||||
}
|
||||
if !w.minimised && behind_key(keys).is_some() && self.desktop.takes_keys_behind(&w.id) {
|
||||
crate::debug_log!("conversation", "pc keys way=behind");
|
||||
return self.desktop.post_key(&w.id, keys).map_err(|code| failed(&code, "keys"));
|
||||
}
|
||||
if (self.game)().is_some() || self.desktop.fullscreen() {
|
||||
crate::debug_log!("conversation", "pc keys way=none (game or full screen)");
|
||||
return Err(refused(NOT_IN_FRONT_NOW));
|
||||
}
|
||||
crate::debug_log!("conversation", "pc keys way=front");
|
||||
let restored = if w.minimised { self.cmd(&args(&["restore", "--window-id", &w.id]), None, "keys").map(|_| ()) } else { Ok(()) };
|
||||
let pressed = restored
|
||||
.and_then(|_| self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys"))
|
||||
.and_then(|_| self.press(w, keys));
|
||||
if let Some(back) = front {
|
||||
if self.desktop.run(&args(&["focus-window", "--window-id", &back]), None, self.stop).is_err() {
|
||||
crate::warn_log!("conversation", "Warning: PC control could not bring back the window that was in front");
|
||||
}
|
||||
}
|
||||
pressed.map(|_| ())
|
||||
}
|
||||
|
||||
/// `press` in window `w`, which is in front. agent-desktop cannot find some Store apps' process
|
||||
/// by the window (the new Calculator: APP_NOT_FOUND, nothing sent), so then the keys go to the
|
||||
/// window in front while it is still `w`.
|
||||
fn press(&self, w: &Win, keys: &str) -> Result<(), Fail> {
|
||||
match self.desktop.run(&args(&["press", keys, "--window-id", &w.id]), None, self.stop) {
|
||||
Err(code) if code == "APP_NOT_FOUND" && self.desktop.foreground().as_deref() == Some(w.id.as_str()) => {
|
||||
crate::debug_log!("conversation", "pc keys press=in-front (window's process not found)");
|
||||
self.cmd(&args(&["press", keys]), None, "keys").map(|_| ())
|
||||
}
|
||||
r => r.map(|_| ()).map_err(|code| failed(&code, "keys")),
|
||||
}
|
||||
}
|
||||
|
||||
fn cmd(&self, args: &[String], stdin: Option<&str>, action: &str) -> Result<Value, Fail> {
|
||||
self.desktop.run(args, stdin, self.stop).map_err(|code| failed(&code, action))
|
||||
}
|
||||
@@ -1056,6 +1174,99 @@ mod win {
|
||||
}
|
||||
}
|
||||
|
||||
/// A window by agent-desktop's id (`w-<handle>`).
|
||||
fn handle(id: &str) -> Option<windows_sys::Win32::Foundation::HWND> {
|
||||
use windows_sys::Win32::UI::WindowsAndMessaging::IsWindow;
|
||||
let h = id.strip_prefix("w-")?.parse::<usize>().ok().filter(|h| *h > 0)? as windows_sys::Win32::Foundation::HWND;
|
||||
// SAFETY: a plain query on a handle value.
|
||||
(unsafe { IsWindow(h) } != 0).then_some(h)
|
||||
}
|
||||
|
||||
fn class_of(h: windows_sys::Win32::Foundation::HWND) -> String {
|
||||
use windows_sys::Win32::UI::WindowsAndMessaging::GetClassNameW;
|
||||
let mut buf = [0u16; 256];
|
||||
// SAFETY: the buffer is sized as passed.
|
||||
let n = unsafe { GetClassNameW(h, buf.as_mut_ptr(), buf.len() as i32) };
|
||||
String::from_utf16_lossy(&buf[..n.max(0) as usize])
|
||||
}
|
||||
|
||||
pub fn foreground() -> Option<String> {
|
||||
use windows_sys::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
|
||||
// SAFETY: a plain query.
|
||||
let h = unsafe { GetForegroundWindow() };
|
||||
(!h.is_null()).then(|| format!("w-{}", h as usize))
|
||||
}
|
||||
|
||||
/// The window in front covers its monitor without a title bar or sizing border (not the
|
||||
/// desktop), or Windows reports a full-screen Direct3D program.
|
||||
pub fn fullscreen() -> bool {
|
||||
use windows_sys::Win32::Foundation::RECT;
|
||||
use windows_sys::Win32::Graphics::Gdi::{GetMonitorInfoW, MonitorFromWindow, MONITORINFO, MONITOR_DEFAULTTONEAREST};
|
||||
use windows_sys::Win32::UI::Shell::{SHQueryUserNotificationState, QUNS_RUNNING_D3D_FULL_SCREEN};
|
||||
use windows_sys::Win32::UI::WindowsAndMessaging::{GetForegroundWindow, GetWindowLongPtrW, GetWindowRect, GWL_STYLE, WS_CAPTION, WS_THICKFRAME};
|
||||
// SAFETY: plain queries on the window in front, into sized locals.
|
||||
unsafe {
|
||||
let mut state = 0;
|
||||
if SHQueryUserNotificationState(&mut state) == 0 && state == QUNS_RUNNING_D3D_FULL_SCREEN {
|
||||
return true;
|
||||
}
|
||||
let h = GetForegroundWindow();
|
||||
if h.is_null() || matches!(class_of(h).as_str(), "Progman" | "WorkerW") {
|
||||
return false;
|
||||
}
|
||||
let mut r: RECT = std::mem::zeroed();
|
||||
let mut mi = MONITORINFO { cbSize: std::mem::size_of::<MONITORINFO>() as u32, ..Default::default() };
|
||||
if GetWindowRect(h, &mut r) == 0 || GetMonitorInfoW(MonitorFromWindow(h, MONITOR_DEFAULTTONEAREST), &mut mi) == 0 {
|
||||
return false;
|
||||
}
|
||||
let m = mi.rcMonitor;
|
||||
let style = GetWindowLongPtrW(h, GWL_STYLE) as u32;
|
||||
r.left <= m.left && r.top <= m.top && r.right >= m.right && r.bottom >= m.bottom
|
||||
&& style & WS_CAPTION != WS_CAPTION && style & WS_THICKFRAME == 0
|
||||
}
|
||||
}
|
||||
|
||||
/// Window `id`'s one text control shown (`text_control`), if it has exactly one. Windows does
|
||||
/// not tell which control of a window behind has the focus (a thread behind has none), so keys
|
||||
/// go behind only where there is no doubt where they land.
|
||||
pub fn text_control_of(id: &str) -> Option<windows_sys::Win32::Foundation::HWND> {
|
||||
use windows_sys::Win32::Foundation::{HWND, LPARAM};
|
||||
use windows_sys::Win32::UI::WindowsAndMessaging::{EnumChildWindows, IsWindowVisible};
|
||||
unsafe extern "system" fn each(child: HWND, found: LPARAM) -> i32 {
|
||||
// SAFETY: `found` is the vector below, alive for the enumeration.
|
||||
let found = unsafe { &mut *(found as *mut Vec<HWND>) };
|
||||
if unsafe { IsWindowVisible(child) } != 0 && super::text_control(&class_of(child)) {
|
||||
found.push(child);
|
||||
}
|
||||
1
|
||||
}
|
||||
let h = handle(id)?;
|
||||
let mut found: Vec<HWND> = vec![];
|
||||
// SAFETY: the callback only reads the children and writes the vector it is given.
|
||||
unsafe { EnumChildWindows(h, Some(each), &mut found as *mut Vec<HWND> as LPARAM) };
|
||||
(found.len() == 1).then(|| found[0])
|
||||
}
|
||||
|
||||
/// Key down and key up for `vk` posted to `to`; the program's message loop makes the character.
|
||||
pub fn post_key(to: windows_sys::Win32::Foundation::HWND, vk: u16) -> Result<(), String> {
|
||||
use windows_sys::Win32::UI::Input::KeyboardAndMouse::{MapVirtualKeyW, MAPVK_VK_TO_VSC};
|
||||
use windows_sys::Win32::UI::WindowsAndMessaging::{PostMessageW, WM_KEYDOWN, WM_KEYUP};
|
||||
// Arrows, Home, End, Page Up and Down, Delete: the extended keys.
|
||||
let extended = matches!(vk, 0x21..=0x28 | 0x2E);
|
||||
// SAFETY: plain message posts to a window handle.
|
||||
unsafe {
|
||||
let scan = MapVirtualKeyW(vk as u32, MAPVK_VK_TO_VSC) as isize;
|
||||
let down = 1 | (scan << 16) | if extended { 1 << 24 } else { 0 };
|
||||
let up = down | (1 << 30) | (1 << 31);
|
||||
if PostMessageW(to, WM_KEYDOWN, vk as usize, down) == 0 || PostMessageW(to, WM_KEYUP, vk as usize, up) == 0 {
|
||||
return Err("ACTION_FAILED".into());
|
||||
}
|
||||
}
|
||||
// Let the program take the keys before the window is read again.
|
||||
std::thread::sleep(std::time::Duration::from_millis(150));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn end_hotkey(thread: u32) {
|
||||
use windows_sys::Win32::UI::WindowsAndMessaging::{PostThreadMessageW, WM_QUIT};
|
||||
// SAFETY: a quit message to the hotkey's own thread.
|
||||
@@ -1085,6 +1296,14 @@ mod tests {
|
||||
/// A password field in every reading, past the elements shown; `get` on it is refused as
|
||||
/// the real agent-desktop refuses it (a secure field's identity is not unique).
|
||||
secure_ref: Option<&'static str>,
|
||||
/// The window in front (`focus-window` moves it), a full-screen window in front, and
|
||||
/// whether windows take keys behind.
|
||||
front: Mutex<Option<String>>,
|
||||
fullscreen: bool,
|
||||
behind: bool,
|
||||
/// agent-desktop cannot find the window's process by its identity (the new Calculator, a
|
||||
/// Store app): `press --window-id` is APP_NOT_FOUND, nothing sent.
|
||||
unnamed: bool,
|
||||
}
|
||||
|
||||
impl Fake {
|
||||
@@ -1094,7 +1313,16 @@ mod tests {
|
||||
{"id": "w-2", "title": "Vault", "app_name": "KeePassXC.exe", "pid": 11, "is_focused": true, "accessible": true},
|
||||
{"id": "w-3", "title": "Calc", "app_name": "CalculatorApp.exe", "pid": 12, "is_focused": false, "minimized": true}])])),
|
||||
processes: vec!["explorer.exe".into()], elevated: false, secure: false, fail: None,
|
||||
calls: Mutex::new(vec![]), launched: json!({"app": "notepad", "pid": 10}), secure_ref: None}
|
||||
calls: Mutex::new(vec![]), launched: json!({"app": "notepad", "pid": 10}), secure_ref: None,
|
||||
front: Mutex::new(Some("w-2".into())), fullscreen: false, behind: false, unnamed: false}
|
||||
}
|
||||
fn front(&self) -> Option<String> {
|
||||
self.front.lock().unwrap().clone()
|
||||
}
|
||||
/// The calls with their arguments, from the first named `from`.
|
||||
fn calls_from(&self, from: &str) -> Vec<Vec<String>> {
|
||||
let calls = self.calls.lock().unwrap();
|
||||
calls.iter().map(|(a, _)| a.clone()).skip_while(|a| a[0] != from).collect()
|
||||
}
|
||||
fn commands(&self) -> Vec<String> {
|
||||
self.calls.lock().unwrap().iter().map(|(a, _)| a[0].clone()).collect()
|
||||
@@ -1119,6 +1347,9 @@ mod tests {
|
||||
if args[0] == "get" && self.secure_ref == Some(args[1].as_str()) {
|
||||
return Err("AMBIGUOUS_TARGET".into());
|
||||
}
|
||||
if args[0] == "press" && self.unnamed && args.iter().any(|a| a == "--window-id") {
|
||||
return Err("APP_NOT_FOUND".into());
|
||||
}
|
||||
Ok(match args[0].as_str() {
|
||||
"list-windows" => {
|
||||
let mut w = self.windows.lock().unwrap();
|
||||
@@ -1134,6 +1365,11 @@ mod tests {
|
||||
}
|
||||
"get" => json!({"property": "states", "value": if self.secure { json!(["secure"]) } else { json!([]) }}),
|
||||
"launch" => self.launched.clone(),
|
||||
// restore activates the window (SW_RESTORE), as focus-window does.
|
||||
"focus-window" | "restore" => {
|
||||
*self.front.lock().unwrap() = Some(args[2].clone());
|
||||
json!({"disposition": {"delivery": "delivered_unverified"}})
|
||||
}
|
||||
_ => json!({"disposition": {"delivery": "delivered_unverified"}}),
|
||||
})
|
||||
}
|
||||
@@ -1143,6 +1379,22 @@ mod tests {
|
||||
fn processes(&self) -> Vec<String> {
|
||||
self.processes.clone()
|
||||
}
|
||||
fn foreground(&self) -> Option<String> {
|
||||
self.front()
|
||||
}
|
||||
fn fullscreen(&self) -> bool {
|
||||
self.fullscreen
|
||||
}
|
||||
fn takes_keys_behind(&self, _id: &str) -> bool {
|
||||
self.behind
|
||||
}
|
||||
fn post_key(&self, id: &str, key: &str) -> Result<(), String> {
|
||||
self.calls.lock().unwrap().push((vec!["post".into(), id.into(), key.into()], None));
|
||||
match self.fail {
|
||||
Some(("post", code)) => Err(code.into()),
|
||||
_ => Ok(()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn settings() -> Settings {
|
||||
@@ -1377,7 +1629,7 @@ mod tests {
|
||||
let (pc, fake, _) = pc_with(s, Fake::new());
|
||||
let o = pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "ctrl+s"})));
|
||||
assert_eq!(o.reply["ok"], true, "{o:?}");
|
||||
assert_eq!(fake.commands(), ["list-windows", "restore", "focus-window", "press", "snapshot"]);
|
||||
assert_eq!(fake.commands(), ["list-windows", "restore", "focus-window", "press", "focus-window", "snapshot"]);
|
||||
let calls = fake.calls.lock().unwrap();
|
||||
assert_eq!(calls[3].0, args(&["press", "ctrl+s", "--window-id", "w-3"]));
|
||||
assert!(calls.iter().all(|(a, _)| !a.iter().any(|x| x == "--force")));
|
||||
@@ -1450,6 +1702,142 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn keys_auto(fake: Fake) -> (Pc, &'static Fake, &'static Stop) {
|
||||
let mut s = settings();
|
||||
s.pc_control.type_text = Level::Auto;
|
||||
pc_with(s, fake)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn one_key_without_modifiers_goes_behind_a_window_that_takes_it_and_nothing_moves() {
|
||||
for key in ["a", "7", "enter", "Tab", "space", "escape", "backspace", "forwarddelete", "left", "pagedown", "f5"] {
|
||||
let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() });
|
||||
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key})));
|
||||
assert_eq!(o.reply["ok"], true, "{key}: {o:?}");
|
||||
assert_eq!(fake.commands(), ["list-windows", "post", "snapshot"], "{key}");
|
||||
assert_eq!(fake.calls_from("post")[0][1], "w-1", "{key}");
|
||||
assert_eq!(fake.front().as_deref(), Some("w-2"), "{key}: the window in front stays");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn combinations_other_keys_and_windows_not_taking_keys_behind_go_in_front_and_back() {
|
||||
let in_front_and_back = |fake: &Fake, key: &str| {
|
||||
assert_eq!(fake.calls_from("focus-window"), [
|
||||
args(&["focus-window", "--window-id", "w-1"]),
|
||||
args(&["press", key, "--window-id", "w-1"]),
|
||||
args(&["focus-window", "--window-id", "w-2"]),
|
||||
args(&["snapshot", "--window-id", "w-1", "--compact", "--timeout-ms", SNAPSHOT_MS])], "{key}");
|
||||
assert!(!fake.commands().contains(&"post".to_owned()), "{key}: never both ways");
|
||||
assert_eq!(fake.front().as_deref(), Some("w-2"), "{key}: the window in front is back");
|
||||
};
|
||||
for key in ["ctrl+s", "shift+tab", "alt+f", "cmd+d", "f10", "shift"] {
|
||||
let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() });
|
||||
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key}))).reply["ok"], true, "{key}");
|
||||
in_front_and_back(fake, key);
|
||||
}
|
||||
// A window that does not take keys behind (a Store app, Chrome) gets even one letter in front.
|
||||
let (pc, fake, _) = keys_auto(Fake::new());
|
||||
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))).reply["ok"], true);
|
||||
in_front_and_back(fake, "a");
|
||||
// A failed press still brings the window in front back, and is not tried behind.
|
||||
let (pc, fake, _) = keys_auto(Fake { fail: Some(("press", "STALE_REF")), ..Fake::new() });
|
||||
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"})));
|
||||
assert_eq!(o.reply, json!({"ok": false, "error": NOT_FRONT}));
|
||||
assert_eq!(fake.calls_from("focus-window").last().unwrap(), &args(&["focus-window", "--window-id", "w-2"]));
|
||||
assert_eq!(fake.front().as_deref(), Some("w-2"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keys_to_a_minimised_window_bring_back_the_window_that_was_in_front_and_wait_for_no_game() {
|
||||
// Restoring brings the window forward, so it is part of the moment in front.
|
||||
let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() });
|
||||
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "ctrl+a"}))).reply["ok"], true);
|
||||
assert_eq!(fake.calls_from("restore"), [
|
||||
args(&["restore", "--window-id", "w-3"]),
|
||||
args(&["focus-window", "--window-id", "w-3"]),
|
||||
args(&["press", "ctrl+a", "--window-id", "w-3"]),
|
||||
args(&["focus-window", "--window-id", "w-2"]),
|
||||
args(&["snapshot", "--window-id", "w-3", "--compact", "--timeout-ms", SNAPSHOT_MS])]);
|
||||
assert_eq!(fake.front().as_deref(), Some("w-2"), "the window in front is back");
|
||||
// During a game or full screen a minimised window is not even restored.
|
||||
let (pc, fake, _) = keys_auto(Fake { behind: true, fullscreen: true, ..Fake::new() });
|
||||
let o = pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "a"})));
|
||||
assert_eq!(o.reply, json!({"ok": false, "error": NOT_IN_FRONT_NOW}));
|
||||
assert!(!fake.commands().contains(&"restore".to_owned()), "{:?}", fake.commands());
|
||||
assert_eq!(fake.front().as_deref(), Some("w-2"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_store_app_agent_desktop_cannot_name_gets_its_keys_once_it_is_in_front() {
|
||||
// Brought in front, and already in front: the key is pressed once, in that window.
|
||||
for already in [false, true] {
|
||||
let fake = Fake { unnamed: true, ..Fake::new() };
|
||||
if already {
|
||||
*fake.front.lock().unwrap() = Some("w-1".into());
|
||||
}
|
||||
let (pc, fake, _) = keys_auto(fake);
|
||||
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "1"})));
|
||||
assert_eq!(o.reply["ok"], true, "{o:?}");
|
||||
let presses: Vec<Vec<String>> = fake.calls_from("list-windows").into_iter().filter(|a| a[0] == "press").collect();
|
||||
assert_eq!(presses.last().unwrap(), &args(&["press", "1"]), "{already}");
|
||||
assert_eq!(fake.front().as_deref(), Some(if already { "w-1" } else { "w-2" }), "{already}: the window in front is back");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_window_already_in_front_gets_its_keys_there_and_nothing_moves() {
|
||||
let fake = Fake::new();
|
||||
*fake.front.lock().unwrap() = Some("w-1".into());
|
||||
let (pc, fake, _) = keys_auto(fake);
|
||||
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "ctrl+s"}))).reply["ok"], true);
|
||||
assert_eq!(fake.commands(), ["list-windows", "press", "snapshot"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keys_are_never_brought_in_front_during_a_game_or_full_screen_but_go_behind() {
|
||||
let game = |fake: Fake| {
|
||||
let mut s = settings();
|
||||
s.pc_control.type_text = Level::Auto;
|
||||
let fake: &'static Fake = Box::leak(Box::new(fake));
|
||||
(Pc::new(s, Box::new(fake), Box::leak(Box::new(Stop::new())), Box::new(|| Some("My Shooter".into()))), fake)
|
||||
};
|
||||
let full = |fake: Fake| {
|
||||
let (pc, fake, _) = keys_auto(Fake { fullscreen: true, ..fake });
|
||||
(pc, fake)
|
||||
};
|
||||
for (i, make) in [&game as &dyn Fn(Fake) -> (Pc, &'static Fake), &full].into_iter().enumerate() {
|
||||
for (key, behind) in [("ctrl+s", true), ("a", false)] {
|
||||
let (pc, fake) = make(Fake { behind, ..Fake::new() });
|
||||
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key})));
|
||||
assert_eq!(o.reply, json!({"ok": false, "error": NOT_IN_FRONT_NOW}), "{i} {key}");
|
||||
assert_eq!(fake.commands(), ["list-windows"], "{i} {key}: no key sent");
|
||||
}
|
||||
let (pc, fake) = make(Fake { behind: true, ..Fake::new() });
|
||||
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))).reply["ok"], true, "{i}");
|
||||
assert_eq!(fake.commands(), ["list-windows", "post", "snapshot"], "{i}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_key_not_taken_behind_is_not_pressed_again_in_front() {
|
||||
let (pc, fake, _) = keys_auto(Fake { behind: true, fail: Some(("post", "ACTION_FAILED")), ..Fake::new() });
|
||||
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"})));
|
||||
assert_eq!(o.reply["ok"], false);
|
||||
assert_eq!(fake.commands(), ["list-windows", "post"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_text_controls_take_keys_behind() {
|
||||
for c in ["Edit", "RichEdit20W", "RICHEDIT50W", "RichEditD2DPT", "Scintilla"] {
|
||||
assert!(text_control(c), "{c}");
|
||||
}
|
||||
for c in ["Notepad", "CalcFrame", "Button", "Windows.UI.Core.CoreWindow", "Chrome_RenderWidgetHostHWND",
|
||||
"Microsoft.UI.Content.DesktopChildSiteBridge", "MozillaWindowClass", "edit"] {
|
||||
assert!(!text_control(c), "{c}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_action_that_closes_its_window_is_done_with_nothing_left_to_read() {
|
||||
let mut s = settings();
|
||||
|
||||
160
app/tests/pc_keys_windows.rs
Normal file
160
app/tests/pc_keys_windows.rs
Normal file
@@ -0,0 +1,160 @@
|
||||
//! PC control's keys on a real Windows desktop (admin.spec.md, "PC control", "Keys"): with another
|
||||
//! window in front, letters and Enter reach Notepad behind it and the window in front stays; a
|
||||
//! combination and the Calculator's keys go in front for a moment and the window in front comes
|
||||
//! back. Runs only on Windows with `JARVIS_TEST_PC_KEYS=1` and agent-desktop at
|
||||
//! `JARVIS_AGENT_DESKTOP` (it opens Notepad, Character Map and the Calculator); elsewhere it skips.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use jarvis_app::pc::{Cli, Desktop, Pc, Stop};
|
||||
use jarvis_app::settings::{Level, Settings};
|
||||
use jarvis_app::worker_client::PcAction;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
const USER: &str = "42";
|
||||
|
||||
fn act(pc: &Pc, action: &str, extra: Value) -> Value {
|
||||
let mut v = json!({"id": "k1", "action": action, "user": USER, "via": "voice"});
|
||||
v.as_object_mut().unwrap().extend(extra.as_object().unwrap().clone());
|
||||
let a: PcAction = serde_json::from_value(v).unwrap();
|
||||
let t = Instant::now();
|
||||
let o = pc.act(&a);
|
||||
println!("{action} {extra}: {} in {} ms", o.code, t.elapsed().as_millis());
|
||||
o.reply
|
||||
}
|
||||
|
||||
/// Every element's name and value of a reading, joined.
|
||||
fn text(reply: &Value) -> String {
|
||||
reply["item"]["elements"].as_array().into_iter().flatten()
|
||||
.map(|e| format!("{} {}", e["name"].as_str().unwrap_or(""), e["value"].as_str().unwrap_or(""))).collect::<Vec<_>>().join(" | ")
|
||||
}
|
||||
|
||||
/// Every element's value of a reading, joined: the typed text without the window's labels (a label
|
||||
/// such as "Ctrl+Shift+X" contains "hi").
|
||||
fn values(reply: &Value) -> String {
|
||||
reply["item"]["elements"].as_array().into_iter().flatten()
|
||||
.filter_map(|e| e["value"].as_str()).collect::<Vec<_>>().join(" | ")
|
||||
}
|
||||
|
||||
/// Closed however the test ends (by name too: a program may hand over to another process).
|
||||
struct Closed(Vec<String>, Vec<std::process::Child>);
|
||||
|
||||
impl Drop for Closed {
|
||||
fn drop(&mut self) {
|
||||
for p in &self.0 {
|
||||
let _ = std::process::Command::new("taskkill").args(["/f", "/im", p]).output();
|
||||
}
|
||||
for c in &mut self.1 {
|
||||
let _ = c.kill();
|
||||
let _ = c.wait();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Minimise window `id` ("w-<handle>") as the person would.
|
||||
#[cfg(windows)]
|
||||
fn minimise(id: &str) {
|
||||
let h: usize = id.trim_start_matches("w-").parse().unwrap();
|
||||
// SAFETY: a plain window call on a handle from list-windows.
|
||||
unsafe { windows_sys::Win32::UI::WindowsAndMessaging::ShowWindow(h as _, windows_sys::Win32::UI::WindowsAndMessaging::SW_MINIMIZE) };
|
||||
}
|
||||
#[cfg(not(windows))]
|
||||
fn minimise(_id: &str) {}
|
||||
|
||||
fn wait_window(pc: &Pc, f: impl Fn(&Value) -> bool) -> String {
|
||||
let until = Instant::now() + Duration::from_secs(15);
|
||||
loop {
|
||||
let list = act(pc, "windows", json!({}));
|
||||
if let Some(w) = list["item"]["windows"].as_array().into_iter().flatten().find(|w| f(w)) {
|
||||
return w["id"].as_str().unwrap().to_owned();
|
||||
}
|
||||
assert!(Instant::now() < until, "the window did not show");
|
||||
std::thread::sleep(Duration::from_millis(500));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keys_go_behind_or_in_front_for_a_moment_and_the_window_in_front_stays() {
|
||||
if std::env::var("JARVIS_TEST_PC_KEYS").is_err() {
|
||||
eprintln!("skipped: set JARVIS_TEST_PC_KEYS=1 and JARVIS_AGENT_DESKTOP");
|
||||
return;
|
||||
}
|
||||
let exe = PathBuf::from(std::env::var("JARVIS_AGENT_DESKTOP").expect("JARVIS_AGENT_DESKTOP"));
|
||||
let data = std::env::temp_dir().join(format!("jarvis-pc-keys-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&data).unwrap();
|
||||
let mut s = Settings::default();
|
||||
s.pc_control.allowed_users = vec![USER.parse().unwrap()];
|
||||
s.pc_control.type_text = Level::Auto;
|
||||
s.pc_control.run = Level::Auto;
|
||||
let stop: &'static Stop = Box::leak(Box::new(Stop::new()));
|
||||
let desktop = Cli::new(&exe, &data);
|
||||
let probe = Cli::new(&exe, &data.join("probe"));
|
||||
let pc = Pc::new(s, Box::new(desktop), stop, Box::new(|| None));
|
||||
// Only programs this test starts are closed: none of them may run before it.
|
||||
let ours = ["notepad.exe", "charmap.exe", "calculatorapp.exe", "win32calc.exe"];
|
||||
let running: Vec<String> = probe.processes().into_iter().filter(|p| ours.contains(&p.as_str())).collect();
|
||||
assert!(running.is_empty(), "close these first: {running:?}");
|
||||
let mut closed = Closed(ours.iter().map(|p| p.to_string()).collect(), vec![]);
|
||||
|
||||
let file = data.join("pckeys.txt");
|
||||
std::fs::write(&file, "").unwrap();
|
||||
closed.1.push(std::process::Command::new("notepad.exe").arg(&file).spawn().unwrap());
|
||||
let notepad = wait_window(&pc, |w| w["title"].as_str().is_some_and(|t| t.contains("pckeys")));
|
||||
closed.1.push(std::process::Command::new("charmap.exe").spawn().unwrap());
|
||||
let front = wait_window(&pc, |w| w["program"] == "charmap");
|
||||
std::thread::sleep(Duration::from_secs(1));
|
||||
let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop);
|
||||
println!("notepad {notepad}; in front {front}: {:?}", probe.foreground());
|
||||
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "Character Map is in front");
|
||||
assert!(probe.takes_keys_behind(¬epad), "Notepad takes keys behind");
|
||||
|
||||
// Letters and Enter behind Notepad: they arrive and the window in front stays.
|
||||
for k in ["h", "i", "enter", "o", "k"] {
|
||||
let r = act(&pc, "keys", json!({"window": notepad, "keys": k}));
|
||||
assert_eq!(r["ok"], true, "{k}: {r}");
|
||||
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "{k}: the window in front stays");
|
||||
}
|
||||
let read = act(&pc, "read", json!({"window": notepad}));
|
||||
println!("notepad after behind: {}", text(&read).chars().take(300).collect::<String>());
|
||||
assert!(values(&read).contains("hi") && values(&read).contains("ok"), "the letters reached Notepad behind");
|
||||
|
||||
// A combination goes in front for a moment, then the window in front is back.
|
||||
let r = act(&pc, "keys", json!({"window": notepad, "keys": "ctrl+a"}));
|
||||
assert_eq!(r["ok"], true, "ctrl+a: {r}");
|
||||
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "ctrl+a: the window in front is back");
|
||||
let r = act(&pc, "keys", json!({"window": notepad, "keys": "x"}));
|
||||
assert_eq!(r["ok"], true, "x: {r}");
|
||||
let after = values(&act(&pc, "read", json!({"window": notepad})));
|
||||
println!("notepad after ctrl+a, x: {after:?}");
|
||||
assert!(after.split(" | ").any(|v| v == "x") && !after.contains("hi"), "ctrl+a selected the text and x replaced it");
|
||||
|
||||
// A minimised window: restored in the moment in front, and the window in front is back.
|
||||
minimise(¬epad);
|
||||
std::thread::sleep(Duration::from_millis(500));
|
||||
let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop);
|
||||
let r = act(&pc, "keys", json!({"window": notepad, "keys": "ctrl+a"}));
|
||||
assert_eq!(r["ok"], true, "minimised ctrl+a: {r}");
|
||||
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "minimised ctrl+a: the window in front is back");
|
||||
let r = act(&pc, "keys", json!({"window": notepad, "keys": "y"}));
|
||||
assert_eq!(r["ok"], true, "y: {r}");
|
||||
let after = values(&act(&pc, "read", json!({"window": notepad})));
|
||||
println!("notepad after minimised, ctrl+a, y: {after:?}");
|
||||
assert!(after.split(" | ").any(|v| v == "y") && !after.split(" | ").any(|v| v == "x"), "ctrl+a reached the restored Notepad");
|
||||
|
||||
// The Calculator: in front for a moment when it does not take keys behind (the Store app).
|
||||
let calc = act(&pc, "run", json!({"program": "calc"}));
|
||||
let calc = calc["item"]["window"]["id"].as_str().expect("the Calculator's window").to_owned();
|
||||
std::thread::sleep(Duration::from_secs(1));
|
||||
let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop);
|
||||
println!("calculator {calc}; takes keys behind: {}", probe.takes_keys_behind(&calc));
|
||||
for k in ["escape", "1", "2"] {
|
||||
let r = act(&pc, "keys", json!({"window": calc, "keys": k}));
|
||||
assert_eq!(r["ok"], true, "{k}: {r}");
|
||||
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "{k}: the window in front stays or is back");
|
||||
}
|
||||
let shown = text(&act(&pc, "read", json!({"window": calc})));
|
||||
println!("calculator after 1 2: {}", shown.chars().take(300).collect::<String>());
|
||||
assert!(shown.contains("12"), "the Calculator shows 12");
|
||||
let _ = std::fs::remove_dir_all(&data);
|
||||
}
|
||||
@@ -226,6 +226,11 @@ LLM과 STT를 함께 켠 측정(M0 `gamemode_stt`, 게임 없음). STT를 먼저
|
||||
- 프로그램 조작은 [agent-desktop](https://github.com/vercel-labs/agent-desktop)(Apache-2.0, Windows UI Automation)으로 합니다. 앱 설치 파일에 실행 파일(약 4MB)을 함께 넣습니다.
|
||||
- 창의 버튼·입력칸·메뉴 목록을 글자로 읽고, 고른 항목을 직접 누르거나 값을 넣습니다. 화면 좌표를 쓰지 않으므로 해상도·배율·창 위치가 바뀌어도 맞게 누릅니다.
|
||||
- 기본은 마우스 커서를 움직이지 않고 포커스도 가져오지 않습니다. 사용자가 다른 창에서 작업하는 중에도 뒤에 있는 창을 조작할 수 있습니다. 최소화된 창은 조작하지 못하므로 먼저 되돌립니다.
|
||||
- 키 누르기는 섞어 씁니다(2026-11-19 사용자 결정). agent-desktop은 앞에 있는 창에만 키를 보낼 수 있어서, 뒤 창으로 보내는 부분은 앱이 Windows 기능으로 직접 합니다.
|
||||
- 기본은 뒤 창에 바로 보냅니다. 조합키가 아닌 키 하나(글자, 숫자, Enter, Tab, 방향키 등)를 그 창의 글 입력칸에 보내고, 사용자 화면과 포커스는 그대로 둡니다. Windows는 뒤 창의 어느 칸에 포커스가 있는지 알려 주지 않으므로, 글 입력칸이 하나뿐인 창(메모장 등)에만 이렇게 보냅니다.
|
||||
- 조합키(Ctrl·Alt·Shift·Win)이거나, 글 입력칸이 하나뿐인 창이 아닐 때(계산기, 스토어·WinUI 앱, 크롬 계열, 파이어폭스 등)만 그 창을 잠깐 앞으로 가져와 키를 누르고, 원래 앞에 있던 창으로 바로 되돌립니다. 그 순간 사용자가 치던 키가 그 창으로 들어갈 수 있습니다.
|
||||
- 게임 중이거나 전체 화면일 때는 앞으로 가져오지 않고 못 한다고 답합니다.
|
||||
- 처음부터 키와 프로그램을 보고 한 가지 방식만 고릅니다. 한 방식이 안 됐다고 다른 방식으로 다시 누르지 않으므로 같은 키가 두 번 눌리지 않습니다. 결과는 창을 다시 읽어 확인합니다.
|
||||
- 누르거나 넣은 뒤 화면을 다시 읽어 결과를 확인하고, 확인된 것만 "했습니다"라고 말합니다.
|
||||
- Alt+F4, Win+R, Ctrl+Shift+Esc 같은 위험 단축키는 agent-desktop이 기본으로 막습니다. 이 막음을 푸는 옵션(`--force`)은 쓰지 않습니다.
|
||||
- 관리자 권한으로 실행된 창은 Windows가 막아서 조작하지 못합니다. 그때는 못 한다고 답합니다.
|
||||
|
||||
@@ -65,7 +65,7 @@ One tool, `action` picks the operation (read as its first known word):
|
||||
| `read` | `window` | the window's interactive elements: ref, role, name, value, state | view |
|
||||
| `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click |
|
||||
| `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type |
|
||||
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type |
|
||||
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`): behind it when it takes them, else in front for a moment (`admin.spec.md`, "PC control") | type |
|
||||
| `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run |
|
||||
| `close` | `window` | closes the program's window | click |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user