diff --git a/app/src/admin.spec.md b/app/src/admin.spec.md index 8f63f6f..81c51bb 100644 --- a/app/src/admin.spec.md +++ b/app/src/admin.spec.md @@ -230,9 +230,7 @@ agent-desktop and decides who may. private to the person: agent-desktop refuses a state folder others may write to, which a folder outside the user's profile is by inheritance), a 15 s limit per command. A program it starts gets the person's own APPDATA and LOCALAPPDATA back and starts in their user - folder. Actions are headless (no cursor, no focus), except `keys`, which brings the window to the - front first (`focus-window`, then `press`); when Windows does not let it come forward (its - foreground lock), no key is sent ("그 창을 앞으로 가져오지 못해 키를 보내지 않았습니다"). `--force` + folder. Actions are headless (no cursor, no focus), `keys` as below. `--force` (which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc and the like) and the raw mouse and key commands are never used; what the brain sends is checked so nothing in it reads as an option, and typed text goes in through stdin (`set-value --stdin`), never in the arguments. The commands: `list-windows`, `snapshot @@ -243,6 +241,32 @@ agent-desktop and decides who may. 떠 있을 수 있습니다). 창을 다시 읽어 주세요", and with several of it running close-app cannot choose: "그 프로그램이 여러 개 실행 중이라 이 창만 닫을 수 없습니다. 창의 닫기 단추를 눌러 주세요"), `restore`. +- **Keys** (`keys`; user decision 2026-11-19): one way is chosen before any key is sent, from the + keys and the window, and never tried the other way after (a key is never pressed twice): + - **Behind** (the default): one key without Ctrl, Alt, Shift or Win that types or moves (`a`-`z`, + `0`-`9`, `enter`, `tab`, `space`, `escape`, `delete`/`backspace`, `forwarddelete`, the arrows, + `home`, `end`, `pageup`, `pagedown`, `f1`-`f9`, `f11`, `f12`; agent-desktop's names) is posted to + the window's text control (`PostMessage`, key down then key up; the program's own message + loop makes the character) without bringing the window forward: the person's screen and focus + stay as they are. Only for a window with exactly one text control shown (`Edit`, `RichEdit20A`, + `RichEdit20W`, `RICHEDIT50W`, the new Notepad's `RichEditD2DPT`, `Scintilla`), which gets the + keys: Windows does not tell which control of a window behind has the focus (a thread behind + has none; win-ci VM 2026-11-19), so keys go behind only where there is no doubt where they land. + Store and WinUI apps (the new Calculator), Chromium, Electron and Firefox windows have none. + - **In front for a moment**: a combination, any other key, and every key for a window that does + not take keys behind. The window is brought to the front (`focus-window`, then `press`) and the + window that was in front before is brought back right after (`focus-window` on it, also when + `press` failed; one that cannot be brought back is left, with a warning in the log). A key the + person types in that moment may go to the window. When Windows does not let the window come + forward (its foreground lock), no key is sent ("그 창을 앞으로 가져오지 못해 키를 보내지 + 않았습니다"). A window already in front gets the keys there and nothing else moves. + - **Never in front during a game or full screen**: while game mode is on, the window in front + covers its whole monitor without a title bar or sizing border (a video or a program full + screen; not the desktop), or Windows reports a full-screen Direct3D program, keys that need + the window in front are not sent ("게임 중이거나 전체 화면이라 그 창을 앞으로 가져오지 않았습니다. + 키를 보내지 않았습니다"). + - Either way the window is read again after ("Results"); keys a program did not take behind show + there as no change, and the brain does not say they were pressed. - **Windows**: as `list-windows` gives them, without this app's own windows, windows without a title and the programs of `pcControl.screenExclude` (never listed, read or acted on, as for screen viewing); at most 30. `window` is a window id or part of a title or program name (the @@ -294,8 +318,8 @@ agent-desktop and decides who may. - **Activity**: each action but `windows` and `read` is `pc.` with the program's file name (never a title, element name or text), who asked and whether it worked, with the refusal's reason ("Activity"); an `ask` answer is not recorded (nothing was done). -- Logs carry the action, the result code and the time taken; never a title, element name, value or - typed text. +- Logs carry the action, the result code and the time taken (for `keys` also the way: behind or in + front); never a title, element name, value, typed text or key. ## Personal notes (`app/src/notes.rs`) @@ -1087,7 +1111,12 @@ a path with non-ASCII letters) or from a Battle.net game's install folder (a tes turns game mode on with its name and off once it ends. Screen viewing (`tests/screen_windows.rs`, `JARVIS_TEST_SCREEN=1`): the monitors and Notepad's window are listed, the window, each monitor and the whole screen come back as JPEG pictures of their size, and with Notepad excluded it is neither -picked nor listed and its area is black on the whole screen; someone not registered is refused. On +picked nor listed and its area is black on the whole screen; someone not registered is refused. +Keys (`tests/pc_keys_windows.rs`, `JARVIS_TEST_PC_KEYS=1`, agent-desktop at `JARVIS_AGENT_DESKTOP`; +it opens Notepad, Character Map and the Calculator, and refuses to start while any of them runs): +with Character Map in front, letters and Enter reach Notepad behind it and Character Map stays in +front; a combination and the Calculator's keys go in front for a moment and Character Map comes +back. On every system with a fake screen: refusals before any capture, monitors by number, the whole screen as placed with excluded areas black (also at another display scale), an excluded window that moves or is brought back during the capture never shown (taken again, or refused while it keeps @@ -1100,7 +1129,9 @@ anti-cheat refusing every action but reading (its own processes and the person's game detection), the emergency stop refusing and lifted, and the real runner killing a running command when it is pressed and at its time limit, a password field and an administrator window refused before acting, readings capped at 200 elements (interactive first) and cut to 120 -characters, `STALE_REF`'s reason, a minimised window restored and `keys` brought to the front, +characters, `STALE_REF`'s reason, a minimised window restored, `keys` sent behind or in front by the keys and the window (one way +only, the window in front brought back, also after a failed press) and never in front during a +game or full screen, excluded and own windows never listed, `run` reading its new window, the activity naming the program only, the hotkey and shortcut formats, and one pin of agent-desktop in `pc.rs`, `package.sh` and M0. Performance stages (on every system, with a fake diff --git a/app/src/pc.rs b/app/src/pc.rs index 8e0c660..bebffc2 100644 --- a/app/src/pc.rs +++ b/app/src/pc.rs @@ -52,6 +52,7 @@ pub const STOPPED: &str = "비상 정지 중입니다. 단축키를 한 번 더 pub const STALE: &str = "화면이 바뀌었습니다. 다시 읽어 주세요"; const NO_WINDOW: &str = "그 창을 찾지 못했습니다"; const NOT_FRONT: &str = "그 창을 앞으로 가져오지 못해 키를 보내지 않았습니다"; +const NOT_IN_FRONT_NOW: &str = "게임 중이거나 전체 화면이라 그 창을 앞으로 가져오지 않았습니다. 키를 보내지 않았습니다"; const NOT_CLOSED: &str = "프로그램이 아직 닫히지 않았습니다(저장할지 묻는 창이 떠 있을 수 있습니다). 창을 다시 읽어 주세요"; const CLOSE_SEVERAL: &str = "그 프로그램이 여러 개 실행 중이라 이 창만 닫을 수 없습니다. 창의 닫기 단추를 눌러 주세요"; const RUNNING_SEVERAL: &str = "그 프로그램이 이미 여러 개 실행 중이라 새로 열지 못했습니다. 열린 창 중에서 골라 주세요"; @@ -212,6 +213,24 @@ pub trait Desktop: Send + Sync { fn program(&self, name: &str) -> String { name.to_owned() } + /// The window in front now, as agent-desktop names it (`w-`). + fn foreground(&self) -> Option { + None + } + /// Whether the window in front fills its monitor without a frame (not the desktop), or a + /// full-screen Direct3D program runs. + fn fullscreen(&self) -> bool { + false + } + /// Whether window `id` takes keys posted to it behind: it has one text control shown + /// (`text_control`). + fn takes_keys_behind(&self, _id: &str) -> bool { + false + } + /// Post one key (`behind_key`) to window `id`'s text control, without bringing it forward. + fn post_key(&self, _id: &str, _key: &str) -> Result<(), String> { + Err("PLATFORM_NOT_SUPPORTED".into()) + } } /// The real agent-desktop: one process per command, its APPDATA, LOCALAPPDATA and state folder in @@ -340,6 +359,28 @@ impl Desktop for Cli { #[cfg(not(windows))] name.to_owned() } + + #[cfg(windows)] + fn foreground(&self) -> Option { + win::foreground() + } + + #[cfg(windows)] + fn fullscreen(&self) -> bool { + win::fullscreen() + } + + #[cfg(windows)] + fn takes_keys_behind(&self, id: &str) -> bool { + win::text_control_of(id).is_some() + } + + #[cfg(windows)] + fn post_key(&self, id: &str, key: &str) -> Result<(), String> { + let vk = behind_key(key).ok_or_else(|| "INVALID_ARGS".to_owned())?; + let to = win::text_control_of(id).ok_or_else(|| "WINDOW_NOT_FOUND".to_owned())?; + win::post_key(to, vk) + } } // ---- the actions -------------------------------------------------------------------------------- @@ -465,6 +506,42 @@ fn valid_keys(k: Option<&str>) -> Result { if ok { Ok(k) } else { Err(refused("키를 알아듣지 못했습니다 (예: ctrl+s, enter)")) } } +/// The virtual key of `keys` when it can be sent behind: one key without modifiers that types or +/// moves, by agent-desktop's names (`a`-`z` and `0`-`9` as their keys on a US layout); else `None`. +fn behind_key(keys: &str) -> Option { + let k = keys.to_ascii_lowercase(); + let vk = match k.as_str() { + "return" | "enter" => 0x0D, + "escape" | "esc" => 0x1B, + "tab" => 0x09, + "space" => 0x20, + "delete" | "backspace" => 0x08, + "forwarddelete" => 0x2E, + "home" => 0x24, + "end" => 0x23, + "pageup" => 0x21, + "pagedown" => 0x22, + "left" => 0x25, + "up" => 0x26, + "right" => 0x27, + "down" => 0x28, + // F10 is Windows' menu key (a system key), so it goes in front. + f if f.len() <= 3 && f.starts_with('f') => match f[1..].parse::() { + Ok(n @ (1..=9 | 11 | 12)) => 0x6F + n, + _ => return None, + }, + c if c.len() == 1 && c.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit()) => c.as_bytes()[0].to_ascii_uppercase() as u16, + _ => return None, + }; + Some(vk) +} + +/// Whether a control of this class takes keys posted behind: the text controls whose keys Windows +/// gives to the control itself (classic Edit and RichEdit, the new Notepad's, Scintilla). +pub fn text_control(class: &str) -> bool { + ["Edit", "RichEdit20A", "RichEdit20W", "RICHEDIT50W", "RichEditD2DPT", "Scintilla"].contains(&class) +} + /// A program to start by name: no folders, nothing that could be read as an option. fn valid_program(p: Option<&str>) -> Result { let p = p.map(str::trim).filter(|p| !p.is_empty()).ok_or_else(|| refused("실행할 프로그램 이름이 필요합니다"))?; @@ -652,11 +729,7 @@ impl Pc { "type" => { self.cmd(&args(&["set-value", element.unwrap(), "--stdin"]), Some(text), "type")?; } - "keys" => { - // Keys go to the window in front, so it is brought there first. - self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys")?; - self.cmd(&args(&["press", keys.as_deref().unwrap(), "--window-id", &w.id]), None, "keys")?; - } + "keys" => self.keys(&w, keys.as_deref().unwrap())?, _ => { self.cmd(&args(&["close-app", &w.program]), None, "close")?; return Ok(Done::Item(json!({"window": w.view(), "elements": []}))); @@ -669,6 +742,35 @@ impl Pc { } } + /// Press `keys` in window `w` one way, chosen first (admin.spec.md, "Keys"): behind it when it is + /// one plain key and the window takes it so, else in front for a moment and the window that was + /// in front brought back. Never both ways: a key is never pressed twice. + fn keys(&self, w: &Win, keys: &str) -> Result<(), Fail> { + let front = self.desktop.foreground(); + if front.as_deref() == Some(w.id.as_str()) { + crate::debug_log!("conversation", "pc keys way=front-already"); + self.cmd(&args(&["press", keys, "--window-id", &w.id]), None, "keys")?; + return Ok(()); + } + if behind_key(keys).is_some() && self.desktop.takes_keys_behind(&w.id) { + crate::debug_log!("conversation", "pc keys way=behind"); + return self.desktop.post_key(&w.id, keys).map_err(|code| failed(&code, "keys")); + } + if (self.game)().is_some() || self.desktop.fullscreen() { + crate::debug_log!("conversation", "pc keys way=none (game or full screen)"); + return Err(refused(NOT_IN_FRONT_NOW)); + } + crate::debug_log!("conversation", "pc keys way=front"); + let pressed = self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys") + .and_then(|_| self.cmd(&args(&["press", keys, "--window-id", &w.id]), None, "keys")); + if let Some(back) = front { + if self.desktop.run(&args(&["focus-window", "--window-id", &back]), None, self.stop).is_err() { + crate::warn_log!("conversation", "Warning: PC control could not bring back the window that was in front"); + } + } + pressed.map(|_| ()) + } + fn cmd(&self, args: &[String], stdin: Option<&str>, action: &str) -> Result { self.desktop.run(args, stdin, self.stop).map_err(|code| failed(&code, action)) } @@ -1056,6 +1158,99 @@ mod win { } } + /// A window by agent-desktop's id (`w-`). + fn handle(id: &str) -> Option { + use windows_sys::Win32::UI::WindowsAndMessaging::IsWindow; + let h = id.strip_prefix("w-")?.parse::().ok().filter(|h| *h > 0)? as windows_sys::Win32::Foundation::HWND; + // SAFETY: a plain query on a handle value. + (unsafe { IsWindow(h) } != 0).then_some(h) + } + + fn class_of(h: windows_sys::Win32::Foundation::HWND) -> String { + use windows_sys::Win32::UI::WindowsAndMessaging::GetClassNameW; + let mut buf = [0u16; 256]; + // SAFETY: the buffer is sized as passed. + let n = unsafe { GetClassNameW(h, buf.as_mut_ptr(), buf.len() as i32) }; + String::from_utf16_lossy(&buf[..n.max(0) as usize]) + } + + pub fn foreground() -> Option { + use windows_sys::Win32::UI::WindowsAndMessaging::GetForegroundWindow; + // SAFETY: a plain query. + let h = unsafe { GetForegroundWindow() }; + (!h.is_null()).then(|| format!("w-{}", h as usize)) + } + + /// The window in front covers its monitor without a title bar or sizing border (not the + /// desktop), or Windows reports a full-screen Direct3D program. + pub fn fullscreen() -> bool { + use windows_sys::Win32::Foundation::RECT; + use windows_sys::Win32::Graphics::Gdi::{GetMonitorInfoW, MonitorFromWindow, MONITORINFO, MONITOR_DEFAULTTONEAREST}; + use windows_sys::Win32::UI::Shell::{SHQueryUserNotificationState, QUNS_RUNNING_D3D_FULL_SCREEN}; + use windows_sys::Win32::UI::WindowsAndMessaging::{GetForegroundWindow, GetWindowLongPtrW, GetWindowRect, GWL_STYLE, WS_CAPTION, WS_THICKFRAME}; + // SAFETY: plain queries on the window in front, into sized locals. + unsafe { + let mut state = 0; + if SHQueryUserNotificationState(&mut state) == 0 && state == QUNS_RUNNING_D3D_FULL_SCREEN { + return true; + } + let h = GetForegroundWindow(); + if h.is_null() || matches!(class_of(h).as_str(), "Progman" | "WorkerW") { + return false; + } + let mut r: RECT = std::mem::zeroed(); + let mut mi = MONITORINFO { cbSize: std::mem::size_of::() as u32, ..Default::default() }; + if GetWindowRect(h, &mut r) == 0 || GetMonitorInfoW(MonitorFromWindow(h, MONITOR_DEFAULTTONEAREST), &mut mi) == 0 { + return false; + } + let m = mi.rcMonitor; + let style = GetWindowLongPtrW(h, GWL_STYLE) as u32; + r.left <= m.left && r.top <= m.top && r.right >= m.right && r.bottom >= m.bottom + && style & WS_CAPTION != WS_CAPTION && style & WS_THICKFRAME == 0 + } + } + + /// Window `id`'s one text control shown (`text_control`), if it has exactly one. Windows does + /// not tell which control of a window behind has the focus (a thread behind has none), so keys + /// go behind only where there is no doubt where they land. + pub fn text_control_of(id: &str) -> Option { + use windows_sys::Win32::Foundation::{HWND, LPARAM}; + use windows_sys::Win32::UI::WindowsAndMessaging::{EnumChildWindows, IsWindowVisible}; + unsafe extern "system" fn each(child: HWND, found: LPARAM) -> i32 { + // SAFETY: `found` is the vector below, alive for the enumeration. + let found = unsafe { &mut *(found as *mut Vec) }; + if unsafe { IsWindowVisible(child) } != 0 && super::text_control(&class_of(child)) { + found.push(child); + } + 1 + } + let h = handle(id)?; + let mut found: Vec = vec![]; + // SAFETY: the callback only reads the children and writes the vector it is given. + unsafe { EnumChildWindows(h, Some(each), &mut found as *mut Vec as LPARAM) }; + (found.len() == 1).then(|| found[0]) + } + + /// Key down and key up for `vk` posted to `to`; the program's message loop makes the character. + pub fn post_key(to: windows_sys::Win32::Foundation::HWND, vk: u16) -> Result<(), String> { + use windows_sys::Win32::UI::Input::KeyboardAndMouse::{MapVirtualKeyW, MAPVK_VK_TO_VSC}; + use windows_sys::Win32::UI::WindowsAndMessaging::{PostMessageW, WM_KEYDOWN, WM_KEYUP}; + // Arrows, Home, End, Page Up and Down, Delete: the extended keys. + let extended = matches!(vk, 0x21..=0x28 | 0x2E); + // SAFETY: plain message posts to a window handle. + unsafe { + let scan = MapVirtualKeyW(vk as u32, MAPVK_VK_TO_VSC) as isize; + let down = 1 | (scan << 16) | if extended { 1 << 24 } else { 0 }; + let up = down | (1 << 30) | (1 << 31); + if PostMessageW(to, WM_KEYDOWN, vk as usize, down) == 0 || PostMessageW(to, WM_KEYUP, vk as usize, up) == 0 { + return Err("ACTION_FAILED".into()); + } + } + // Let the program take the keys before the window is read again. + std::thread::sleep(std::time::Duration::from_millis(150)); + Ok(()) + } + pub fn end_hotkey(thread: u32) { use windows_sys::Win32::UI::WindowsAndMessaging::{PostThreadMessageW, WM_QUIT}; // SAFETY: a quit message to the hotkey's own thread. @@ -1085,6 +1280,11 @@ mod tests { /// A password field in every reading, past the elements shown; `get` on it is refused as /// the real agent-desktop refuses it (a secure field's identity is not unique). secure_ref: Option<&'static str>, + /// The window in front (`focus-window` moves it), a full-screen window in front, and + /// whether windows take keys behind. + front: Mutex>, + fullscreen: bool, + behind: bool, } impl Fake { @@ -1094,7 +1294,16 @@ mod tests { {"id": "w-2", "title": "Vault", "app_name": "KeePassXC.exe", "pid": 11, "is_focused": true, "accessible": true}, {"id": "w-3", "title": "Calc", "app_name": "CalculatorApp.exe", "pid": 12, "is_focused": false, "minimized": true}])])), processes: vec!["explorer.exe".into()], elevated: false, secure: false, fail: None, - calls: Mutex::new(vec![]), launched: json!({"app": "notepad", "pid": 10}), secure_ref: None} + calls: Mutex::new(vec![]), launched: json!({"app": "notepad", "pid": 10}), secure_ref: None, + front: Mutex::new(Some("w-2".into())), fullscreen: false, behind: false} + } + fn front(&self) -> Option { + self.front.lock().unwrap().clone() + } + /// The calls with their arguments, from the first named `from`. + fn calls_from(&self, from: &str) -> Vec> { + let calls = self.calls.lock().unwrap(); + calls.iter().map(|(a, _)| a.clone()).skip_while(|a| a[0] != from).collect() } fn commands(&self) -> Vec { self.calls.lock().unwrap().iter().map(|(a, _)| a[0].clone()).collect() @@ -1134,6 +1343,10 @@ mod tests { } "get" => json!({"property": "states", "value": if self.secure { json!(["secure"]) } else { json!([]) }}), "launch" => self.launched.clone(), + "focus-window" => { + *self.front.lock().unwrap() = Some(args[2].clone()); + json!({"disposition": {"delivery": "delivered_unverified"}}) + } _ => json!({"disposition": {"delivery": "delivered_unverified"}}), }) } @@ -1143,6 +1356,22 @@ mod tests { fn processes(&self) -> Vec { self.processes.clone() } + fn foreground(&self) -> Option { + self.front() + } + fn fullscreen(&self) -> bool { + self.fullscreen + } + fn takes_keys_behind(&self, _id: &str) -> bool { + self.behind + } + fn post_key(&self, id: &str, key: &str) -> Result<(), String> { + self.calls.lock().unwrap().push((vec!["post".into(), id.into(), key.into()], None)); + match self.fail { + Some(("post", code)) => Err(code.into()), + _ => Ok(()), + } + } } fn settings() -> Settings { @@ -1377,7 +1606,7 @@ mod tests { let (pc, fake, _) = pc_with(s, Fake::new()); let o = pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "ctrl+s"}))); assert_eq!(o.reply["ok"], true, "{o:?}"); - assert_eq!(fake.commands(), ["list-windows", "restore", "focus-window", "press", "snapshot"]); + assert_eq!(fake.commands(), ["list-windows", "restore", "focus-window", "press", "focus-window", "snapshot"]); let calls = fake.calls.lock().unwrap(); assert_eq!(calls[3].0, args(&["press", "ctrl+s", "--window-id", "w-3"])); assert!(calls.iter().all(|(a, _)| !a.iter().any(|x| x == "--force"))); @@ -1450,6 +1679,105 @@ mod tests { } } + fn keys_auto(fake: Fake) -> (Pc, &'static Fake, &'static Stop) { + let mut s = settings(); + s.pc_control.type_text = Level::Auto; + pc_with(s, fake) + } + + #[test] + fn one_key_without_modifiers_goes_behind_a_window_that_takes_it_and_nothing_moves() { + for key in ["a", "7", "enter", "Tab", "space", "escape", "backspace", "forwarddelete", "left", "pagedown", "f5"] { + let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() }); + let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key}))); + assert_eq!(o.reply["ok"], true, "{key}: {o:?}"); + assert_eq!(fake.commands(), ["list-windows", "post", "snapshot"], "{key}"); + assert_eq!(fake.calls_from("post")[0][1], "w-1", "{key}"); + assert_eq!(fake.front().as_deref(), Some("w-2"), "{key}: the window in front stays"); + } + } + + #[test] + fn combinations_other_keys_and_windows_not_taking_keys_behind_go_in_front_and_back() { + let in_front_and_back = |fake: &Fake, key: &str| { + assert_eq!(fake.calls_from("focus-window"), [ + args(&["focus-window", "--window-id", "w-1"]), + args(&["press", key, "--window-id", "w-1"]), + args(&["focus-window", "--window-id", "w-2"]), + args(&["snapshot", "--window-id", "w-1", "--compact", "--timeout-ms", SNAPSHOT_MS])], "{key}"); + assert!(!fake.commands().contains(&"post".to_owned()), "{key}: never both ways"); + assert_eq!(fake.front().as_deref(), Some("w-2"), "{key}: the window in front is back"); + }; + for key in ["ctrl+s", "shift+tab", "alt+f", "cmd+d", "f10", "shift"] { + let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() }); + assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key}))).reply["ok"], true, "{key}"); + in_front_and_back(fake, key); + } + // A window that does not take keys behind (a Store app, Chrome) gets even one letter in front. + let (pc, fake, _) = keys_auto(Fake::new()); + assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))).reply["ok"], true); + in_front_and_back(fake, "a"); + // A failed press still brings the window in front back, and is not tried behind. + let (pc, fake, _) = keys_auto(Fake { fail: Some(("press", "STALE_REF")), ..Fake::new() }); + let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))); + assert_eq!(o.reply, json!({"ok": false, "error": NOT_FRONT})); + assert_eq!(fake.calls_from("focus-window").last().unwrap(), &args(&["focus-window", "--window-id", "w-2"])); + assert_eq!(fake.front().as_deref(), Some("w-2")); + } + + #[test] + fn a_window_already_in_front_gets_its_keys_there_and_nothing_moves() { + let fake = Fake::new(); + *fake.front.lock().unwrap() = Some("w-1".into()); + let (pc, fake, _) = keys_auto(fake); + assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "ctrl+s"}))).reply["ok"], true); + assert_eq!(fake.commands(), ["list-windows", "press", "snapshot"]); + } + + #[test] + fn keys_are_never_brought_in_front_during_a_game_or_full_screen_but_go_behind() { + let game = |fake: Fake| { + let mut s = settings(); + s.pc_control.type_text = Level::Auto; + let fake: &'static Fake = Box::leak(Box::new(fake)); + (Pc::new(s, Box::new(fake), Box::leak(Box::new(Stop::new())), Box::new(|| Some("My Shooter".into()))), fake) + }; + let full = |fake: Fake| { + let (pc, fake, _) = keys_auto(Fake { fullscreen: true, ..fake }); + (pc, fake) + }; + for (i, make) in [&game as &dyn Fn(Fake) -> (Pc, &'static Fake), &full].into_iter().enumerate() { + for (key, behind) in [("ctrl+s", true), ("a", false)] { + let (pc, fake) = make(Fake { behind, ..Fake::new() }); + let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key}))); + assert_eq!(o.reply, json!({"ok": false, "error": NOT_IN_FRONT_NOW}), "{i} {key}"); + assert_eq!(fake.commands(), ["list-windows"], "{i} {key}: no key sent"); + } + let (pc, fake) = make(Fake { behind: true, ..Fake::new() }); + assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))).reply["ok"], true, "{i}"); + assert_eq!(fake.commands(), ["list-windows", "post", "snapshot"], "{i}"); + } + } + + #[test] + fn a_key_not_taken_behind_is_not_pressed_again_in_front() { + let (pc, fake, _) = keys_auto(Fake { behind: true, fail: Some(("post", "ACTION_FAILED")), ..Fake::new() }); + let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))); + assert_eq!(o.reply["ok"], false); + assert_eq!(fake.commands(), ["list-windows", "post"]); + } + + #[test] + fn only_text_controls_take_keys_behind() { + for c in ["Edit", "RichEdit20W", "RICHEDIT50W", "RichEditD2DPT", "Scintilla"] { + assert!(text_control(c), "{c}"); + } + for c in ["Notepad", "CalcFrame", "Button", "Windows.UI.Core.CoreWindow", "Chrome_RenderWidgetHostHWND", + "Microsoft.UI.Content.DesktopChildSiteBridge", "MozillaWindowClass", "edit"] { + assert!(!text_control(c), "{c}"); + } + } + #[test] fn an_action_that_closes_its_window_is_done_with_nothing_left_to_read() { let mut s = settings(); diff --git a/app/tests/pc_keys_windows.rs b/app/tests/pc_keys_windows.rs new file mode 100644 index 0000000..6d03e6a --- /dev/null +++ b/app/tests/pc_keys_windows.rs @@ -0,0 +1,130 @@ +//! PC control's keys on a real Windows desktop (admin.spec.md, "PC control", "Keys"): with another +//! window in front, letters and Enter reach Notepad behind it and the window in front stays; a +//! combination and the Calculator's keys go in front for a moment and the window in front comes +//! back. Runs only on Windows with `JARVIS_TEST_PC_KEYS=1` and agent-desktop at +//! `JARVIS_AGENT_DESKTOP` (it opens Notepad, Character Map and the Calculator); elsewhere it skips. + +use std::path::PathBuf; +use std::time::{Duration, Instant}; + +use jarvis_app::pc::{Cli, Desktop, Pc, Stop}; +use jarvis_app::settings::{Level, Settings}; +use jarvis_app::worker_client::PcAction; +use serde_json::{json, Value}; + +const USER: &str = "42"; + +fn act(pc: &Pc, action: &str, extra: Value) -> Value { + let mut v = json!({"id": "k1", "action": action, "user": USER, "via": "voice"}); + v.as_object_mut().unwrap().extend(extra.as_object().unwrap().clone()); + let a: PcAction = serde_json::from_value(v).unwrap(); + let t = Instant::now(); + let o = pc.act(&a); + println!("{action} {extra}: {} in {} ms", o.code, t.elapsed().as_millis()); + o.reply +} + +/// Every element's name and value of a reading, joined. +fn text(reply: &Value) -> String { + reply["item"]["elements"].as_array().into_iter().flatten() + .map(|e| format!("{} {}", e["name"].as_str().unwrap_or(""), e["value"].as_str().unwrap_or(""))).collect::>().join(" | ") +} + +/// Closed however the test ends (by name too: a program may hand over to another process). +struct Closed(Vec, Vec); + +impl Drop for Closed { + fn drop(&mut self) { + for p in &self.0 { + let _ = std::process::Command::new("taskkill").args(["/f", "/im", p]).output(); + } + for c in &mut self.1 { + let _ = c.kill(); + let _ = c.wait(); + } + } +} + +fn wait_window(pc: &Pc, f: impl Fn(&Value) -> bool) -> String { + let until = Instant::now() + Duration::from_secs(15); + loop { + let list = act(pc, "windows", json!({})); + if let Some(w) = list["item"]["windows"].as_array().into_iter().flatten().find(|w| f(w)) { + return w["id"].as_str().unwrap().to_owned(); + } + assert!(Instant::now() < until, "the window did not show"); + std::thread::sleep(Duration::from_millis(500)); + } +} + +#[test] +fn keys_go_behind_or_in_front_for_a_moment_and_the_window_in_front_stays() { + if std::env::var("JARVIS_TEST_PC_KEYS").is_err() { + eprintln!("skipped: set JARVIS_TEST_PC_KEYS=1 and JARVIS_AGENT_DESKTOP"); + return; + } + let exe = PathBuf::from(std::env::var("JARVIS_AGENT_DESKTOP").expect("JARVIS_AGENT_DESKTOP")); + let data = std::env::temp_dir().join(format!("jarvis-pc-keys-{}", std::process::id())); + std::fs::create_dir_all(&data).unwrap(); + let mut s = Settings::default(); + s.pc_control.allowed_users = vec![USER.parse().unwrap()]; + s.pc_control.type_text = Level::Auto; + s.pc_control.run = Level::Auto; + let stop: &'static Stop = Box::leak(Box::new(Stop::new())); + let desktop = Cli::new(&exe, &data); + let probe = Cli::new(&exe, &data.join("probe")); + let pc = Pc::new(s, Box::new(desktop), stop, Box::new(|| None)); + // Only programs this test starts are closed: none of them may run before it. + let ours = ["notepad.exe", "charmap.exe", "calculatorapp.exe", "win32calc.exe"]; + let running: Vec = probe.processes().into_iter().filter(|p| ours.contains(&p.as_str())).collect(); + assert!(running.is_empty(), "close these first: {running:?}"); + let mut closed = Closed(ours.iter().map(|p| p.to_string()).collect(), vec![]); + + let file = data.join("pckeys.txt"); + std::fs::write(&file, "").unwrap(); + closed.1.push(std::process::Command::new("notepad.exe").arg(&file).spawn().unwrap()); + let notepad = wait_window(&pc, |w| w["title"].as_str().is_some_and(|t| t.contains("pckeys"))); + closed.1.push(std::process::Command::new("charmap.exe").spawn().unwrap()); + let front = wait_window(&pc, |w| w["program"] == "charmap"); + std::thread::sleep(Duration::from_secs(1)); + let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop); + println!("notepad {notepad}; in front {front}: {:?}", probe.foreground()); + assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "Character Map is in front"); + assert!(probe.takes_keys_behind(¬epad), "Notepad takes keys behind"); + + // Letters and Enter behind Notepad: they arrive and the window in front stays. + for k in ["h", "i", "enter", "o", "k"] { + let r = act(&pc, "keys", json!({"window": notepad, "keys": k})); + assert_eq!(r["ok"], true, "{k}: {r}"); + assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "{k}: the window in front stays"); + } + let read = act(&pc, "read", json!({"window": notepad})); + println!("notepad after behind: {}", text(&read).chars().take(300).collect::()); + assert!(text(&read).contains("hi") && text(&read).contains("ok"), "the letters reached Notepad behind"); + + // A combination goes in front for a moment, then the window in front is back. + let r = act(&pc, "keys", json!({"window": notepad, "keys": "ctrl+a"})); + assert_eq!(r["ok"], true, "ctrl+a: {r}"); + assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "ctrl+a: the window in front is back"); + let r = act(&pc, "keys", json!({"window": notepad, "keys": "x"})); + assert_eq!(r["ok"], true, "x: {r}"); + let after = text(&act(&pc, "read", json!({"window": notepad}))); + println!("notepad after ctrl+a, x: {}", after.chars().take(300).collect::()); + assert!(!after.contains("hi"), "ctrl+a selected the text and x replaced it"); + + // The Calculator: in front for a moment when it does not take keys behind (the Store app). + let calc = act(&pc, "run", json!({"program": "calc"})); + let calc = calc["item"]["window"]["id"].as_str().expect("the Calculator's window").to_owned(); + std::thread::sleep(Duration::from_secs(1)); + let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop); + println!("calculator {calc}; takes keys behind: {}", probe.takes_keys_behind(&calc)); + for k in ["escape", "1", "2"] { + let r = act(&pc, "keys", json!({"window": calc, "keys": k})); + assert_eq!(r["ok"], true, "{k}: {r}"); + assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "{k}: the window in front stays or is back"); + } + let shown = text(&act(&pc, "read", json!({"window": calc}))); + println!("calculator after 1 2: {}", shown.chars().take(300).collect::()); + assert!(shown.contains("12"), "the Calculator shows 12"); + let _ = std::fs::remove_dir_all(&data); +} diff --git a/docs/v2/pc-features.md b/docs/v2/pc-features.md index 5e2a447..22d576d 100644 --- a/docs/v2/pc-features.md +++ b/docs/v2/pc-features.md @@ -226,6 +226,11 @@ LLM과 STT를 함께 켠 측정(M0 `gamemode_stt`, 게임 없음). STT를 먼저 - 프로그램 조작은 [agent-desktop](https://github.com/vercel-labs/agent-desktop)(Apache-2.0, Windows UI Automation)으로 합니다. 앱 설치 파일에 실행 파일(약 4MB)을 함께 넣습니다. - 창의 버튼·입력칸·메뉴 목록을 글자로 읽고, 고른 항목을 직접 누르거나 값을 넣습니다. 화면 좌표를 쓰지 않으므로 해상도·배율·창 위치가 바뀌어도 맞게 누릅니다. - 기본은 마우스 커서를 움직이지 않고 포커스도 가져오지 않습니다. 사용자가 다른 창에서 작업하는 중에도 뒤에 있는 창을 조작할 수 있습니다. 최소화된 창은 조작하지 못하므로 먼저 되돌립니다. + - 키 누르기는 섞어 씁니다(2026-11-19 사용자 결정). agent-desktop은 앞에 있는 창에만 키를 보낼 수 있어서, 뒤 창으로 보내는 부분은 앱이 Windows 기능으로 직접 합니다. + - 기본은 뒤 창에 바로 보냅니다. 조합키가 아닌 키 하나(글자, 숫자, Enter, Tab, 방향키 등)를 그 창의 글 입력칸에 보내고, 사용자 화면과 포커스는 그대로 둡니다. Windows는 뒤 창의 어느 칸에 포커스가 있는지 알려 주지 않으므로, 글 입력칸이 하나뿐인 창(메모장 등)에만 이렇게 보냅니다. + - 조합키(Ctrl·Alt·Shift·Win)이거나, 글 입력칸이 하나뿐인 창이 아닐 때(계산기, 스토어·WinUI 앱, 크롬 계열, 파이어폭스 등)만 그 창을 잠깐 앞으로 가져와 키를 누르고, 원래 앞에 있던 창으로 바로 되돌립니다. 그 순간 사용자가 치던 키가 그 창으로 들어갈 수 있습니다. + - 게임 중이거나 전체 화면일 때는 앞으로 가져오지 않고 못 한다고 답합니다. + - 처음부터 키와 프로그램을 보고 한 가지 방식만 고릅니다. 한 방식이 안 됐다고 다른 방식으로 다시 누르지 않으므로 같은 키가 두 번 눌리지 않습니다. 결과는 창을 다시 읽어 확인합니다. - 누르거나 넣은 뒤 화면을 다시 읽어 결과를 확인하고, 확인된 것만 "했습니다"라고 말합니다. - Alt+F4, Win+R, Ctrl+Shift+Esc 같은 위험 단축키는 agent-desktop이 기본으로 막습니다. 이 막음을 푸는 옵션(`--force`)은 쓰지 않습니다. - 관리자 권한으로 실행된 창은 Windows가 막아서 조작하지 못합니다. 그때는 못 한다고 답합니다. diff --git a/src/jarvis/tools/builtin/pc_control.spec.md b/src/jarvis/tools/builtin/pc_control.spec.md index 7c87db6..e3071b7 100644 --- a/src/jarvis/tools/builtin/pc_control.spec.md +++ b/src/jarvis/tools/builtin/pc_control.spec.md @@ -65,7 +65,7 @@ One tool, `action` picks the operation (read as its first known word): | `read` | `window` | the window's interactive elements: ref, role, name, value, state | view | | `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click | | `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type | -| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type | +| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`): behind it when it takes them, else in front for a moment (`admin.spec.md`, "PC control") | type | | `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run | | `close` | `window` | closes the program's window | click |