Compare commits

..

5 Commits

Author SHA1 Message Date
javis-bot
3af881935c chore(app): version 0.1.44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
javis-bot
20bd90f65e fix(v2): pcControl keys to a minimised window bring back the window that was in front
restore activates the window (SW_RESTORE), so restoring before keys made the
window already in front: the window that was in front before was not brought
back, and the game / full-screen check came after the window had moved. Keys
now note the window in front and check for a game first, then restore inside
the moment in front. A minimised window never takes keys behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
javis-bot
9a296a5683 chore(app): version 0.1.43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:55:55 +09:00
javis-bot
5767ffa5ae fix(v2): pcControl keys reach a Store app agent-desktop cannot name by its window
On the game PC agent-desktop's press --window-id answers APP_NOT_FOUND for
the new Calculator ("not found with exact process identity", nothing sent),
though focus-window brings it in front. The keys now go to the window in
front while it is still that window. The Windows test checks Notepad's typed
text by element values only (a label such as Ctrl+Shift+X contains "hi").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:55:19 +09:00
javis-bot
378ccfafad feat(v2): pcControl keys go behind a window when they can, else in front for a moment
User decision 2026-11-19 (mixed way). agent-desktop only presses keys in the
window in front, so the app now picks one way before any key is sent:

- behind: one key without modifiers that types or moves is posted to the
  window's one shown text control (Edit, RichEdit, RichEditD2DPT, Scintilla);
  the person's screen and focus stay. Windows does not tell a background
  thread's focused control (win-ci VM: GetFocus is null even attached), so
  only windows with exactly one text control get keys behind.
- in front for a moment: combinations, other keys and every other window;
  the window that was in front is brought back after (also after a failed
  press). A window already in front gets the keys there.
- never in front while game mode is on or the window in front is full screen.
- never both ways, so a key is never pressed twice.

Specs: admin.spec.md "PC control" (Keys), pc-features.md, pc_control.spec.md.
Tests: six new fake-desktop tests in pc.rs; tests/pc_keys_windows.rs on a real
desktop (win-ci VM: letters behind Notepad with Character Map kept in front,
ctrl+a in front and back, the classic Calculator in front and back).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:41:05 +09:00
9 changed files with 613 additions and 22 deletions

2
app/Cargo.lock generated
View File

@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.42"
version = "0.1.44"
dependencies = [
"async-trait",
"base64 0.22.1",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-app"
version = "0.1.42"
version = "0.1.44"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"

4
app/shell/Cargo.lock generated
View File

@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.42"
version = "0.1.44"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -2666,7 +2666,7 @@ dependencies = [
[[package]]
name = "jarvis-shell"
version = "0.1.42"
version = "0.1.44"
dependencies = [
"jarvis-app",
"serde_json",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-shell"
version = "0.1.42"
version = "0.1.44"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"

View File

@@ -230,9 +230,7 @@ agent-desktop and decides who may.
private to the person: agent-desktop refuses a state folder others may write to, which a folder
outside the user's profile is by inheritance), a 15 s limit per command. A
program it starts gets the person's own APPDATA and LOCALAPPDATA back and starts in their user
folder. Actions are headless (no cursor, no focus), except `keys`, which brings the window to the
front first (`focus-window`, then `press`); when Windows does not let it come forward (its
foreground lock), no key is sent ("그 창을 앞으로 가져오지 못해 키를 보내지 않았습니다"). `--force`
folder. Actions are headless (no cursor, no focus), `keys` as below. `--force`
(which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc and the like) and the raw mouse and key
commands are never used; what the brain sends is checked so nothing in it reads as an option,
and typed text goes in through stdin (`set-value --stdin`), never in the arguments. The commands: `list-windows`, `snapshot
@@ -243,6 +241,38 @@ agent-desktop and decides who may.
떠 있을 수 있습니다). 창을 다시 읽어 주세요", and with several of it running close-app cannot
choose: "그 프로그램이 여러 개 실행 중이라 이 창만 닫을 수 없습니다. 창의 닫기 단추를 눌러 주세요"),
`restore`.
- **Keys** (`keys`; user decision 2026-11-19): one way is chosen before any key is sent, from the
keys and the window, and never tried the other way after (a key is never pressed twice):
- **Behind** (the default): one key without Ctrl, Alt, Shift or Win that types or moves (`a`-`z`,
`0`-`9`, `enter`, `tab`, `space`, `escape`, `delete`/`backspace`, `forwarddelete`, the arrows,
`home`, `end`, `pageup`, `pagedown`, `f1`-`f9`, `f11`, `f12`; agent-desktop's names) is posted to
the window's text control (`PostMessage`, key down then key up; the program's own message
loop makes the character) without bringing the window forward: the person's screen and focus
stay as they are. Only for a window with exactly one text control shown (`Edit`, `RichEdit20A`,
`RichEdit20W`, `RICHEDIT50W`, the new Notepad's `RichEditD2DPT`, `Scintilla`), which gets the
keys: Windows does not tell which control of a window behind has the focus (a thread behind
has none; win-ci VM 2026-11-19), so keys go behind only where there is no doubt where they land.
Store and WinUI apps (the new Calculator), Chromium, Electron and Firefox windows have none.
- **In front for a moment**: a combination, any other key, and every key for a window that does
not take keys behind or is minimised. A minimised window is restored inside the moment
(restoring brings it forward, so the window in front is noted before and brought back after;
the window itself stays restored). The window is brought to the front (`focus-window`, then `press`) and the
window that was in front before is brought back right after (`focus-window` on it, also when
`press` failed; one that cannot be brought back is left, with a warning in the log). For a
window whose process agent-desktop cannot find from the window (the new Calculator, a Store
app: `press --window-id` is `APP_NOT_FOUND` and sends nothing; game PC 2026-11-19) the keys
are pressed without naming the window (nothing was sent before), into the window in front,
only while that is still the window (checked just before). A key the
person types in that moment may go to the window. When Windows does not let the window come
forward (its foreground lock), no key is sent ("그 창을 앞으로 가져오지 못해 키를 보내지
않았습니다"). A window already in front gets the keys there and nothing else moves.
- **Never in front during a game or full screen**: while game mode is on, the window in front
covers its whole monitor without a title bar or sizing border (a video or a program full
screen; not the desktop), or Windows reports a full-screen Direct3D program, keys that need
the window in front are not sent ("게임 중이거나 전체 화면이라 그 창을 앞으로 가져오지 않았습니다.
키를 보내지 않았습니다").
- Either way the window is read again after ("Results"); keys a program did not take behind show
there as no change, and the brain does not say they were pressed.
- **Windows**: as `list-windows` gives them, without this app's own windows, windows without a
title and the programs of `pcControl.screenExclude` (never listed, read or acted on, as for
screen viewing); at most 30. `window` is a window id or part of a title or program name (the
@@ -287,15 +317,16 @@ agent-desktop and decides who may.
interactive elements read afresh after the action (at most 200), so the brain says it was done
only from what it sees; an action that closes its window (a dialog's button, Enter) answers the
window with no elements. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting; for `keys` inside
the moment in front, "Keys").
- **Refused before asking**: the stop, the anti-cheat, an administrator window (a process whose
elevation cannot be read counts as one) and a password field are refused without asking.
`windows` and `read` at `ask` answer `ask: true` like the rest.
- **Activity**: each action but `windows` and `read` is `pc.<action>` with the program's file name
(never a title, element name or text), who asked and whether it worked, with the refusal's reason
("Activity"); an `ask` answer is not recorded (nothing was done).
- Logs carry the action, the result code and the time taken; never a title, element name, value or
typed text.
- Logs carry the action, the result code and the time taken (for `keys` also the way: behind or in
front); never a title, element name, value, typed text or key.
## Personal notes (`app/src/notes.rs`)
@@ -1087,7 +1118,12 @@ a path with non-ASCII letters) or from a Battle.net game's install folder (a tes
turns game mode on with its name and off once it ends. Screen viewing (`tests/screen_windows.rs`,
`JARVIS_TEST_SCREEN=1`): the monitors and Notepad's window are listed, the window, each monitor and
the whole screen come back as JPEG pictures of their size, and with Notepad excluded it is neither
picked nor listed and its area is black on the whole screen; someone not registered is refused. On
picked nor listed and its area is black on the whole screen; someone not registered is refused.
Keys (`tests/pc_keys_windows.rs`, `JARVIS_TEST_PC_KEYS=1`, agent-desktop at `JARVIS_AGENT_DESKTOP`;
it opens Notepad, Character Map and the Calculator, and refuses to start while any of them runs):
with Character Map in front, letters and Enter reach Notepad behind it and Character Map stays in
front; a combination and the Calculator's keys go in front for a moment and Character Map comes
back. On
every system with a fake screen: refusals before any capture, monitors by number, the whole screen
as placed with excluded areas black (also at another display scale), an excluded window that
moves or is brought back during the capture never shown (taken again, or refused while it keeps
@@ -1100,7 +1136,9 @@ anti-cheat refusing every action but reading (its own processes and the person's
game detection), the emergency stop refusing and lifted, and the real runner killing a running
command when it is pressed and at its time limit, a password field and an administrator window
refused before acting, readings capped at 200 elements (interactive first) and cut to 120
characters, `STALE_REF`'s reason, a minimised window restored and `keys` brought to the front,
characters, `STALE_REF`'s reason, a minimised window restored, `keys` sent behind or in front by the keys and the window (one way
only, the window in front brought back, also after a failed press) and never in front during a
game or full screen,
excluded and own windows never listed, `run` reading its new window, the activity naming the
program only, the hotkey and shortcut formats, and one pin of agent-desktop in `pc.rs`,
`package.sh` and M0. Performance stages (on every system, with a fake

View File

@@ -52,6 +52,7 @@ pub const STOPPED: &str = "비상 정지 중입니다. 단축키를 한 번 더
pub const STALE: &str = "화면이 바뀌었습니다. 다시 읽어 주세요";
const NO_WINDOW: &str = "그 창을 찾지 못했습니다";
const NOT_FRONT: &str = "그 창을 앞으로 가져오지 못해 키를 보내지 않았습니다";
const NOT_IN_FRONT_NOW: &str = "게임 중이거나 전체 화면이라 그 창을 앞으로 가져오지 않았습니다. 키를 보내지 않았습니다";
const NOT_CLOSED: &str = "프로그램이 아직 닫히지 않았습니다(저장할지 묻는 창이 떠 있을 수 있습니다). 창을 다시 읽어 주세요";
const CLOSE_SEVERAL: &str = "그 프로그램이 여러 개 실행 중이라 이 창만 닫을 수 없습니다. 창의 닫기 단추를 눌러 주세요";
const RUNNING_SEVERAL: &str = "그 프로그램이 이미 여러 개 실행 중이라 새로 열지 못했습니다. 열린 창 중에서 골라 주세요";
@@ -212,6 +213,24 @@ pub trait Desktop: Send + Sync {
fn program(&self, name: &str) -> String {
name.to_owned()
}
/// The window in front now, as agent-desktop names it (`w-<handle>`).
fn foreground(&self) -> Option<String> {
None
}
/// Whether the window in front fills its monitor without a frame (not the desktop), or a
/// full-screen Direct3D program runs.
fn fullscreen(&self) -> bool {
false
}
/// Whether window `id` takes keys posted to it behind: it has one text control shown
/// (`text_control`).
fn takes_keys_behind(&self, _id: &str) -> bool {
false
}
/// Post one key (`behind_key`) to window `id`'s text control, without bringing it forward.
fn post_key(&self, _id: &str, _key: &str) -> Result<(), String> {
Err("PLATFORM_NOT_SUPPORTED".into())
}
}
/// The real agent-desktop: one process per command, its APPDATA, LOCALAPPDATA and state folder in
@@ -340,6 +359,28 @@ impl Desktop for Cli {
#[cfg(not(windows))]
name.to_owned()
}
#[cfg(windows)]
fn foreground(&self) -> Option<String> {
win::foreground()
}
#[cfg(windows)]
fn fullscreen(&self) -> bool {
win::fullscreen()
}
#[cfg(windows)]
fn takes_keys_behind(&self, id: &str) -> bool {
win::text_control_of(id).is_some()
}
#[cfg(windows)]
fn post_key(&self, id: &str, key: &str) -> Result<(), String> {
let vk = behind_key(key).ok_or_else(|| "INVALID_ARGS".to_owned())?;
let to = win::text_control_of(id).ok_or_else(|| "WINDOW_NOT_FOUND".to_owned())?;
win::post_key(to, vk)
}
}
// ---- the actions --------------------------------------------------------------------------------
@@ -465,6 +506,42 @@ fn valid_keys(k: Option<&str>) -> Result<String, Fail> {
if ok { Ok(k) } else { Err(refused("키를 알아듣지 못했습니다 (예: ctrl+s, enter)")) }
}
/// The virtual key of `keys` when it can be sent behind: one key without modifiers that types or
/// moves, by agent-desktop's names (`a`-`z` and `0`-`9` as their keys on a US layout); else `None`.
fn behind_key(keys: &str) -> Option<u16> {
let k = keys.to_ascii_lowercase();
let vk = match k.as_str() {
"return" | "enter" => 0x0D,
"escape" | "esc" => 0x1B,
"tab" => 0x09,
"space" => 0x20,
"delete" | "backspace" => 0x08,
"forwarddelete" => 0x2E,
"home" => 0x24,
"end" => 0x23,
"pageup" => 0x21,
"pagedown" => 0x22,
"left" => 0x25,
"up" => 0x26,
"right" => 0x27,
"down" => 0x28,
// F10 is Windows' menu key (a system key), so it goes in front.
f if f.len() <= 3 && f.starts_with('f') => match f[1..].parse::<u16>() {
Ok(n @ (1..=9 | 11 | 12)) => 0x6F + n,
_ => return None,
},
c if c.len() == 1 && c.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit()) => c.as_bytes()[0].to_ascii_uppercase() as u16,
_ => return None,
};
Some(vk)
}
/// Whether a control of this class takes keys posted behind: the text controls whose keys Windows
/// gives to the control itself (classic Edit and RichEdit, the new Notepad's, Scintilla).
pub fn text_control(class: &str) -> bool {
["Edit", "RichEdit20A", "RichEdit20W", "RICHEDIT50W", "RichEditD2DPT", "Scintilla"].contains(&class)
}
/// A program to start by name: no folders, nothing that could be read as an option.
fn valid_program(p: Option<&str>) -> Result<String, Fail> {
let p = p.map(str::trim).filter(|p| !p.is_empty()).ok_or_else(|| refused("실행할 프로그램 이름이 필요합니다"))?;
@@ -642,7 +719,8 @@ impl Pc {
if self.stop.on() {
return Err(refused(STOPPED));
}
if w.minimised {
// Keys restore in their moment in front (restoring brings the window forward).
if w.minimised && a.action != "keys" {
self.cmd(&args(&["restore", "--window-id", &w.id]), None, &a.action)?;
}
match a.action.as_str() {
@@ -652,11 +730,7 @@ impl Pc {
"type" => {
self.cmd(&args(&["set-value", element.unwrap(), "--stdin"]), Some(text), "type")?;
}
"keys" => {
// Keys go to the window in front, so it is brought there first.
self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys")?;
self.cmd(&args(&["press", keys.as_deref().unwrap(), "--window-id", &w.id]), None, "keys")?;
}
"keys" => self.keys(&w, keys.as_deref().unwrap())?,
_ => {
self.cmd(&args(&["close-app", &w.program]), None, "close")?;
return Ok(Done::Item(json!({"window": w.view(), "elements": []})));
@@ -669,6 +743,50 @@ impl Pc {
}
}
/// Press `keys` in window `w` one way, chosen first (admin.spec.md, "Keys"): behind it when it is
/// one plain key and the window takes it so, else in front for a moment (a minimised window
/// restored in it) and the window that was in front brought back. Never both ways: a key is
/// never pressed twice.
fn keys(&self, w: &Win, keys: &str) -> Result<(), Fail> {
let front = self.desktop.foreground();
if front.as_deref() == Some(w.id.as_str()) {
crate::debug_log!("conversation", "pc keys way=front-already");
return self.press(w, keys);
}
if !w.minimised && behind_key(keys).is_some() && self.desktop.takes_keys_behind(&w.id) {
crate::debug_log!("conversation", "pc keys way=behind");
return self.desktop.post_key(&w.id, keys).map_err(|code| failed(&code, "keys"));
}
if (self.game)().is_some() || self.desktop.fullscreen() {
crate::debug_log!("conversation", "pc keys way=none (game or full screen)");
return Err(refused(NOT_IN_FRONT_NOW));
}
crate::debug_log!("conversation", "pc keys way=front");
let restored = if w.minimised { self.cmd(&args(&["restore", "--window-id", &w.id]), None, "keys").map(|_| ()) } else { Ok(()) };
let pressed = restored
.and_then(|_| self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys"))
.and_then(|_| self.press(w, keys));
if let Some(back) = front {
if self.desktop.run(&args(&["focus-window", "--window-id", &back]), None, self.stop).is_err() {
crate::warn_log!("conversation", "Warning: PC control could not bring back the window that was in front");
}
}
pressed.map(|_| ())
}
/// `press` in window `w`, which is in front. agent-desktop cannot find some Store apps' process
/// by the window (the new Calculator: APP_NOT_FOUND, nothing sent), so then the keys go to the
/// window in front while it is still `w`.
fn press(&self, w: &Win, keys: &str) -> Result<(), Fail> {
match self.desktop.run(&args(&["press", keys, "--window-id", &w.id]), None, self.stop) {
Err(code) if code == "APP_NOT_FOUND" && self.desktop.foreground().as_deref() == Some(w.id.as_str()) => {
crate::debug_log!("conversation", "pc keys press=in-front (window's process not found)");
self.cmd(&args(&["press", keys]), None, "keys").map(|_| ())
}
r => r.map(|_| ()).map_err(|code| failed(&code, "keys")),
}
}
fn cmd(&self, args: &[String], stdin: Option<&str>, action: &str) -> Result<Value, Fail> {
self.desktop.run(args, stdin, self.stop).map_err(|code| failed(&code, action))
}
@@ -1056,6 +1174,99 @@ mod win {
}
}
/// A window by agent-desktop's id (`w-<handle>`).
fn handle(id: &str) -> Option<windows_sys::Win32::Foundation::HWND> {
use windows_sys::Win32::UI::WindowsAndMessaging::IsWindow;
let h = id.strip_prefix("w-")?.parse::<usize>().ok().filter(|h| *h > 0)? as windows_sys::Win32::Foundation::HWND;
// SAFETY: a plain query on a handle value.
(unsafe { IsWindow(h) } != 0).then_some(h)
}
fn class_of(h: windows_sys::Win32::Foundation::HWND) -> String {
use windows_sys::Win32::UI::WindowsAndMessaging::GetClassNameW;
let mut buf = [0u16; 256];
// SAFETY: the buffer is sized as passed.
let n = unsafe { GetClassNameW(h, buf.as_mut_ptr(), buf.len() as i32) };
String::from_utf16_lossy(&buf[..n.max(0) as usize])
}
pub fn foreground() -> Option<String> {
use windows_sys::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
// SAFETY: a plain query.
let h = unsafe { GetForegroundWindow() };
(!h.is_null()).then(|| format!("w-{}", h as usize))
}
/// The window in front covers its monitor without a title bar or sizing border (not the
/// desktop), or Windows reports a full-screen Direct3D program.
pub fn fullscreen() -> bool {
use windows_sys::Win32::Foundation::RECT;
use windows_sys::Win32::Graphics::Gdi::{GetMonitorInfoW, MonitorFromWindow, MONITORINFO, MONITOR_DEFAULTTONEAREST};
use windows_sys::Win32::UI::Shell::{SHQueryUserNotificationState, QUNS_RUNNING_D3D_FULL_SCREEN};
use windows_sys::Win32::UI::WindowsAndMessaging::{GetForegroundWindow, GetWindowLongPtrW, GetWindowRect, GWL_STYLE, WS_CAPTION, WS_THICKFRAME};
// SAFETY: plain queries on the window in front, into sized locals.
unsafe {
let mut state = 0;
if SHQueryUserNotificationState(&mut state) == 0 && state == QUNS_RUNNING_D3D_FULL_SCREEN {
return true;
}
let h = GetForegroundWindow();
if h.is_null() || matches!(class_of(h).as_str(), "Progman" | "WorkerW") {
return false;
}
let mut r: RECT = std::mem::zeroed();
let mut mi = MONITORINFO { cbSize: std::mem::size_of::<MONITORINFO>() as u32, ..Default::default() };
if GetWindowRect(h, &mut r) == 0 || GetMonitorInfoW(MonitorFromWindow(h, MONITOR_DEFAULTTONEAREST), &mut mi) == 0 {
return false;
}
let m = mi.rcMonitor;
let style = GetWindowLongPtrW(h, GWL_STYLE) as u32;
r.left <= m.left && r.top <= m.top && r.right >= m.right && r.bottom >= m.bottom
&& style & WS_CAPTION != WS_CAPTION && style & WS_THICKFRAME == 0
}
}
/// Window `id`'s one text control shown (`text_control`), if it has exactly one. Windows does
/// not tell which control of a window behind has the focus (a thread behind has none), so keys
/// go behind only where there is no doubt where they land.
pub fn text_control_of(id: &str) -> Option<windows_sys::Win32::Foundation::HWND> {
use windows_sys::Win32::Foundation::{HWND, LPARAM};
use windows_sys::Win32::UI::WindowsAndMessaging::{EnumChildWindows, IsWindowVisible};
unsafe extern "system" fn each(child: HWND, found: LPARAM) -> i32 {
// SAFETY: `found` is the vector below, alive for the enumeration.
let found = unsafe { &mut *(found as *mut Vec<HWND>) };
if unsafe { IsWindowVisible(child) } != 0 && super::text_control(&class_of(child)) {
found.push(child);
}
1
}
let h = handle(id)?;
let mut found: Vec<HWND> = vec![];
// SAFETY: the callback only reads the children and writes the vector it is given.
unsafe { EnumChildWindows(h, Some(each), &mut found as *mut Vec<HWND> as LPARAM) };
(found.len() == 1).then(|| found[0])
}
/// Key down and key up for `vk` posted to `to`; the program's message loop makes the character.
pub fn post_key(to: windows_sys::Win32::Foundation::HWND, vk: u16) -> Result<(), String> {
use windows_sys::Win32::UI::Input::KeyboardAndMouse::{MapVirtualKeyW, MAPVK_VK_TO_VSC};
use windows_sys::Win32::UI::WindowsAndMessaging::{PostMessageW, WM_KEYDOWN, WM_KEYUP};
// Arrows, Home, End, Page Up and Down, Delete: the extended keys.
let extended = matches!(vk, 0x21..=0x28 | 0x2E);
// SAFETY: plain message posts to a window handle.
unsafe {
let scan = MapVirtualKeyW(vk as u32, MAPVK_VK_TO_VSC) as isize;
let down = 1 | (scan << 16) | if extended { 1 << 24 } else { 0 };
let up = down | (1 << 30) | (1 << 31);
if PostMessageW(to, WM_KEYDOWN, vk as usize, down) == 0 || PostMessageW(to, WM_KEYUP, vk as usize, up) == 0 {
return Err("ACTION_FAILED".into());
}
}
// Let the program take the keys before the window is read again.
std::thread::sleep(std::time::Duration::from_millis(150));
Ok(())
}
pub fn end_hotkey(thread: u32) {
use windows_sys::Win32::UI::WindowsAndMessaging::{PostThreadMessageW, WM_QUIT};
// SAFETY: a quit message to the hotkey's own thread.
@@ -1085,6 +1296,14 @@ mod tests {
/// A password field in every reading, past the elements shown; `get` on it is refused as
/// the real agent-desktop refuses it (a secure field's identity is not unique).
secure_ref: Option<&'static str>,
/// The window in front (`focus-window` moves it), a full-screen window in front, and
/// whether windows take keys behind.
front: Mutex<Option<String>>,
fullscreen: bool,
behind: bool,
/// agent-desktop cannot find the window's process by its identity (the new Calculator, a
/// Store app): `press --window-id` is APP_NOT_FOUND, nothing sent.
unnamed: bool,
}
impl Fake {
@@ -1094,7 +1313,16 @@ mod tests {
{"id": "w-2", "title": "Vault", "app_name": "KeePassXC.exe", "pid": 11, "is_focused": true, "accessible": true},
{"id": "w-3", "title": "Calc", "app_name": "CalculatorApp.exe", "pid": 12, "is_focused": false, "minimized": true}])])),
processes: vec!["explorer.exe".into()], elevated: false, secure: false, fail: None,
calls: Mutex::new(vec![]), launched: json!({"app": "notepad", "pid": 10}), secure_ref: None}
calls: Mutex::new(vec![]), launched: json!({"app": "notepad", "pid": 10}), secure_ref: None,
front: Mutex::new(Some("w-2".into())), fullscreen: false, behind: false, unnamed: false}
}
fn front(&self) -> Option<String> {
self.front.lock().unwrap().clone()
}
/// The calls with their arguments, from the first named `from`.
fn calls_from(&self, from: &str) -> Vec<Vec<String>> {
let calls = self.calls.lock().unwrap();
calls.iter().map(|(a, _)| a.clone()).skip_while(|a| a[0] != from).collect()
}
fn commands(&self) -> Vec<String> {
self.calls.lock().unwrap().iter().map(|(a, _)| a[0].clone()).collect()
@@ -1119,6 +1347,9 @@ mod tests {
if args[0] == "get" && self.secure_ref == Some(args[1].as_str()) {
return Err("AMBIGUOUS_TARGET".into());
}
if args[0] == "press" && self.unnamed && args.iter().any(|a| a == "--window-id") {
return Err("APP_NOT_FOUND".into());
}
Ok(match args[0].as_str() {
"list-windows" => {
let mut w = self.windows.lock().unwrap();
@@ -1134,6 +1365,11 @@ mod tests {
}
"get" => json!({"property": "states", "value": if self.secure { json!(["secure"]) } else { json!([]) }}),
"launch" => self.launched.clone(),
// restore activates the window (SW_RESTORE), as focus-window does.
"focus-window" | "restore" => {
*self.front.lock().unwrap() = Some(args[2].clone());
json!({"disposition": {"delivery": "delivered_unverified"}})
}
_ => json!({"disposition": {"delivery": "delivered_unverified"}}),
})
}
@@ -1143,6 +1379,22 @@ mod tests {
fn processes(&self) -> Vec<String> {
self.processes.clone()
}
fn foreground(&self) -> Option<String> {
self.front()
}
fn fullscreen(&self) -> bool {
self.fullscreen
}
fn takes_keys_behind(&self, _id: &str) -> bool {
self.behind
}
fn post_key(&self, id: &str, key: &str) -> Result<(), String> {
self.calls.lock().unwrap().push((vec!["post".into(), id.into(), key.into()], None));
match self.fail {
Some(("post", code)) => Err(code.into()),
_ => Ok(()),
}
}
}
fn settings() -> Settings {
@@ -1377,7 +1629,7 @@ mod tests {
let (pc, fake, _) = pc_with(s, Fake::new());
let o = pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "ctrl+s"})));
assert_eq!(o.reply["ok"], true, "{o:?}");
assert_eq!(fake.commands(), ["list-windows", "restore", "focus-window", "press", "snapshot"]);
assert_eq!(fake.commands(), ["list-windows", "restore", "focus-window", "press", "focus-window", "snapshot"]);
let calls = fake.calls.lock().unwrap();
assert_eq!(calls[3].0, args(&["press", "ctrl+s", "--window-id", "w-3"]));
assert!(calls.iter().all(|(a, _)| !a.iter().any(|x| x == "--force")));
@@ -1450,6 +1702,142 @@ mod tests {
}
}
fn keys_auto(fake: Fake) -> (Pc, &'static Fake, &'static Stop) {
let mut s = settings();
s.pc_control.type_text = Level::Auto;
pc_with(s, fake)
}
#[test]
fn one_key_without_modifiers_goes_behind_a_window_that_takes_it_and_nothing_moves() {
for key in ["a", "7", "enter", "Tab", "space", "escape", "backspace", "forwarddelete", "left", "pagedown", "f5"] {
let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() });
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key})));
assert_eq!(o.reply["ok"], true, "{key}: {o:?}");
assert_eq!(fake.commands(), ["list-windows", "post", "snapshot"], "{key}");
assert_eq!(fake.calls_from("post")[0][1], "w-1", "{key}");
assert_eq!(fake.front().as_deref(), Some("w-2"), "{key}: the window in front stays");
}
}
#[test]
fn combinations_other_keys_and_windows_not_taking_keys_behind_go_in_front_and_back() {
let in_front_and_back = |fake: &Fake, key: &str| {
assert_eq!(fake.calls_from("focus-window"), [
args(&["focus-window", "--window-id", "w-1"]),
args(&["press", key, "--window-id", "w-1"]),
args(&["focus-window", "--window-id", "w-2"]),
args(&["snapshot", "--window-id", "w-1", "--compact", "--timeout-ms", SNAPSHOT_MS])], "{key}");
assert!(!fake.commands().contains(&"post".to_owned()), "{key}: never both ways");
assert_eq!(fake.front().as_deref(), Some("w-2"), "{key}: the window in front is back");
};
for key in ["ctrl+s", "shift+tab", "alt+f", "cmd+d", "f10", "shift"] {
let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() });
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key}))).reply["ok"], true, "{key}");
in_front_and_back(fake, key);
}
// A window that does not take keys behind (a Store app, Chrome) gets even one letter in front.
let (pc, fake, _) = keys_auto(Fake::new());
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))).reply["ok"], true);
in_front_and_back(fake, "a");
// A failed press still brings the window in front back, and is not tried behind.
let (pc, fake, _) = keys_auto(Fake { fail: Some(("press", "STALE_REF")), ..Fake::new() });
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"})));
assert_eq!(o.reply, json!({"ok": false, "error": NOT_FRONT}));
assert_eq!(fake.calls_from("focus-window").last().unwrap(), &args(&["focus-window", "--window-id", "w-2"]));
assert_eq!(fake.front().as_deref(), Some("w-2"));
}
#[test]
fn keys_to_a_minimised_window_bring_back_the_window_that_was_in_front_and_wait_for_no_game() {
// Restoring brings the window forward, so it is part of the moment in front.
let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() });
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "ctrl+a"}))).reply["ok"], true);
assert_eq!(fake.calls_from("restore"), [
args(&["restore", "--window-id", "w-3"]),
args(&["focus-window", "--window-id", "w-3"]),
args(&["press", "ctrl+a", "--window-id", "w-3"]),
args(&["focus-window", "--window-id", "w-2"]),
args(&["snapshot", "--window-id", "w-3", "--compact", "--timeout-ms", SNAPSHOT_MS])]);
assert_eq!(fake.front().as_deref(), Some("w-2"), "the window in front is back");
// During a game or full screen a minimised window is not even restored.
let (pc, fake, _) = keys_auto(Fake { behind: true, fullscreen: true, ..Fake::new() });
let o = pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "a"})));
assert_eq!(o.reply, json!({"ok": false, "error": NOT_IN_FRONT_NOW}));
assert!(!fake.commands().contains(&"restore".to_owned()), "{:?}", fake.commands());
assert_eq!(fake.front().as_deref(), Some("w-2"));
}
#[test]
fn a_store_app_agent_desktop_cannot_name_gets_its_keys_once_it_is_in_front() {
// Brought in front, and already in front: the key is pressed once, in that window.
for already in [false, true] {
let fake = Fake { unnamed: true, ..Fake::new() };
if already {
*fake.front.lock().unwrap() = Some("w-1".into());
}
let (pc, fake, _) = keys_auto(fake);
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "1"})));
assert_eq!(o.reply["ok"], true, "{o:?}");
let presses: Vec<Vec<String>> = fake.calls_from("list-windows").into_iter().filter(|a| a[0] == "press").collect();
assert_eq!(presses.last().unwrap(), &args(&["press", "1"]), "{already}");
assert_eq!(fake.front().as_deref(), Some(if already { "w-1" } else { "w-2" }), "{already}: the window in front is back");
}
}
#[test]
fn a_window_already_in_front_gets_its_keys_there_and_nothing_moves() {
let fake = Fake::new();
*fake.front.lock().unwrap() = Some("w-1".into());
let (pc, fake, _) = keys_auto(fake);
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "ctrl+s"}))).reply["ok"], true);
assert_eq!(fake.commands(), ["list-windows", "press", "snapshot"]);
}
#[test]
fn keys_are_never_brought_in_front_during_a_game_or_full_screen_but_go_behind() {
let game = |fake: Fake| {
let mut s = settings();
s.pc_control.type_text = Level::Auto;
let fake: &'static Fake = Box::leak(Box::new(fake));
(Pc::new(s, Box::new(fake), Box::leak(Box::new(Stop::new())), Box::new(|| Some("My Shooter".into()))), fake)
};
let full = |fake: Fake| {
let (pc, fake, _) = keys_auto(Fake { fullscreen: true, ..fake });
(pc, fake)
};
for (i, make) in [&game as &dyn Fn(Fake) -> (Pc, &'static Fake), &full].into_iter().enumerate() {
for (key, behind) in [("ctrl+s", true), ("a", false)] {
let (pc, fake) = make(Fake { behind, ..Fake::new() });
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": key})));
assert_eq!(o.reply, json!({"ok": false, "error": NOT_IN_FRONT_NOW}), "{i} {key}");
assert_eq!(fake.commands(), ["list-windows"], "{i} {key}: no key sent");
}
let (pc, fake) = make(Fake { behind: true, ..Fake::new() });
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"}))).reply["ok"], true, "{i}");
assert_eq!(fake.commands(), ["list-windows", "post", "snapshot"], "{i}");
}
}
#[test]
fn a_key_not_taken_behind_is_not_pressed_again_in_front() {
let (pc, fake, _) = keys_auto(Fake { behind: true, fail: Some(("post", "ACTION_FAILED")), ..Fake::new() });
let o = pc.act(&action("keys", OWNER, json!({"window": "notepad", "keys": "a"})));
assert_eq!(o.reply["ok"], false);
assert_eq!(fake.commands(), ["list-windows", "post"]);
}
#[test]
fn only_text_controls_take_keys_behind() {
for c in ["Edit", "RichEdit20W", "RICHEDIT50W", "RichEditD2DPT", "Scintilla"] {
assert!(text_control(c), "{c}");
}
for c in ["Notepad", "CalcFrame", "Button", "Windows.UI.Core.CoreWindow", "Chrome_RenderWidgetHostHWND",
"Microsoft.UI.Content.DesktopChildSiteBridge", "MozillaWindowClass", "edit"] {
assert!(!text_control(c), "{c}");
}
}
#[test]
fn an_action_that_closes_its_window_is_done_with_nothing_left_to_read() {
let mut s = settings();

View File

@@ -0,0 +1,160 @@
//! PC control's keys on a real Windows desktop (admin.spec.md, "PC control", "Keys"): with another
//! window in front, letters and Enter reach Notepad behind it and the window in front stays; a
//! combination and the Calculator's keys go in front for a moment and the window in front comes
//! back. Runs only on Windows with `JARVIS_TEST_PC_KEYS=1` and agent-desktop at
//! `JARVIS_AGENT_DESKTOP` (it opens Notepad, Character Map and the Calculator); elsewhere it skips.
use std::path::PathBuf;
use std::time::{Duration, Instant};
use jarvis_app::pc::{Cli, Desktop, Pc, Stop};
use jarvis_app::settings::{Level, Settings};
use jarvis_app::worker_client::PcAction;
use serde_json::{json, Value};
const USER: &str = "42";
fn act(pc: &Pc, action: &str, extra: Value) -> Value {
let mut v = json!({"id": "k1", "action": action, "user": USER, "via": "voice"});
v.as_object_mut().unwrap().extend(extra.as_object().unwrap().clone());
let a: PcAction = serde_json::from_value(v).unwrap();
let t = Instant::now();
let o = pc.act(&a);
println!("{action} {extra}: {} in {} ms", o.code, t.elapsed().as_millis());
o.reply
}
/// Every element's name and value of a reading, joined.
fn text(reply: &Value) -> String {
reply["item"]["elements"].as_array().into_iter().flatten()
.map(|e| format!("{} {}", e["name"].as_str().unwrap_or(""), e["value"].as_str().unwrap_or(""))).collect::<Vec<_>>().join(" | ")
}
/// Every element's value of a reading, joined: the typed text without the window's labels (a label
/// such as "Ctrl+Shift+X" contains "hi").
fn values(reply: &Value) -> String {
reply["item"]["elements"].as_array().into_iter().flatten()
.filter_map(|e| e["value"].as_str()).collect::<Vec<_>>().join(" | ")
}
/// Closed however the test ends (by name too: a program may hand over to another process).
struct Closed(Vec<String>, Vec<std::process::Child>);
impl Drop for Closed {
fn drop(&mut self) {
for p in &self.0 {
let _ = std::process::Command::new("taskkill").args(["/f", "/im", p]).output();
}
for c in &mut self.1 {
let _ = c.kill();
let _ = c.wait();
}
}
}
/// Minimise window `id` ("w-<handle>") as the person would.
#[cfg(windows)]
fn minimise(id: &str) {
let h: usize = id.trim_start_matches("w-").parse().unwrap();
// SAFETY: a plain window call on a handle from list-windows.
unsafe { windows_sys::Win32::UI::WindowsAndMessaging::ShowWindow(h as _, windows_sys::Win32::UI::WindowsAndMessaging::SW_MINIMIZE) };
}
#[cfg(not(windows))]
fn minimise(_id: &str) {}
fn wait_window(pc: &Pc, f: impl Fn(&Value) -> bool) -> String {
let until = Instant::now() + Duration::from_secs(15);
loop {
let list = act(pc, "windows", json!({}));
if let Some(w) = list["item"]["windows"].as_array().into_iter().flatten().find(|w| f(w)) {
return w["id"].as_str().unwrap().to_owned();
}
assert!(Instant::now() < until, "the window did not show");
std::thread::sleep(Duration::from_millis(500));
}
}
#[test]
fn keys_go_behind_or_in_front_for_a_moment_and_the_window_in_front_stays() {
if std::env::var("JARVIS_TEST_PC_KEYS").is_err() {
eprintln!("skipped: set JARVIS_TEST_PC_KEYS=1 and JARVIS_AGENT_DESKTOP");
return;
}
let exe = PathBuf::from(std::env::var("JARVIS_AGENT_DESKTOP").expect("JARVIS_AGENT_DESKTOP"));
let data = std::env::temp_dir().join(format!("jarvis-pc-keys-{}", std::process::id()));
std::fs::create_dir_all(&data).unwrap();
let mut s = Settings::default();
s.pc_control.allowed_users = vec![USER.parse().unwrap()];
s.pc_control.type_text = Level::Auto;
s.pc_control.run = Level::Auto;
let stop: &'static Stop = Box::leak(Box::new(Stop::new()));
let desktop = Cli::new(&exe, &data);
let probe = Cli::new(&exe, &data.join("probe"));
let pc = Pc::new(s, Box::new(desktop), stop, Box::new(|| None));
// Only programs this test starts are closed: none of them may run before it.
let ours = ["notepad.exe", "charmap.exe", "calculatorapp.exe", "win32calc.exe"];
let running: Vec<String> = probe.processes().into_iter().filter(|p| ours.contains(&p.as_str())).collect();
assert!(running.is_empty(), "close these first: {running:?}");
let mut closed = Closed(ours.iter().map(|p| p.to_string()).collect(), vec![]);
let file = data.join("pckeys.txt");
std::fs::write(&file, "").unwrap();
closed.1.push(std::process::Command::new("notepad.exe").arg(&file).spawn().unwrap());
let notepad = wait_window(&pc, |w| w["title"].as_str().is_some_and(|t| t.contains("pckeys")));
closed.1.push(std::process::Command::new("charmap.exe").spawn().unwrap());
let front = wait_window(&pc, |w| w["program"] == "charmap");
std::thread::sleep(Duration::from_secs(1));
let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop);
println!("notepad {notepad}; in front {front}: {:?}", probe.foreground());
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "Character Map is in front");
assert!(probe.takes_keys_behind(&notepad), "Notepad takes keys behind");
// Letters and Enter behind Notepad: they arrive and the window in front stays.
for k in ["h", "i", "enter", "o", "k"] {
let r = act(&pc, "keys", json!({"window": notepad, "keys": k}));
assert_eq!(r["ok"], true, "{k}: {r}");
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "{k}: the window in front stays");
}
let read = act(&pc, "read", json!({"window": notepad}));
println!("notepad after behind: {}", text(&read).chars().take(300).collect::<String>());
assert!(values(&read).contains("hi") && values(&read).contains("ok"), "the letters reached Notepad behind");
// A combination goes in front for a moment, then the window in front is back.
let r = act(&pc, "keys", json!({"window": notepad, "keys": "ctrl+a"}));
assert_eq!(r["ok"], true, "ctrl+a: {r}");
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "ctrl+a: the window in front is back");
let r = act(&pc, "keys", json!({"window": notepad, "keys": "x"}));
assert_eq!(r["ok"], true, "x: {r}");
let after = values(&act(&pc, "read", json!({"window": notepad})));
println!("notepad after ctrl+a, x: {after:?}");
assert!(after.split(" | ").any(|v| v == "x") && !after.contains("hi"), "ctrl+a selected the text and x replaced it");
// A minimised window: restored in the moment in front, and the window in front is back.
minimise(&notepad);
std::thread::sleep(Duration::from_millis(500));
let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop);
let r = act(&pc, "keys", json!({"window": notepad, "keys": "ctrl+a"}));
assert_eq!(r["ok"], true, "minimised ctrl+a: {r}");
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "minimised ctrl+a: the window in front is back");
let r = act(&pc, "keys", json!({"window": notepad, "keys": "y"}));
assert_eq!(r["ok"], true, "y: {r}");
let after = values(&act(&pc, "read", json!({"window": notepad})));
println!("notepad after minimised, ctrl+a, y: {after:?}");
assert!(after.split(" | ").any(|v| v == "y") && !after.split(" | ").any(|v| v == "x"), "ctrl+a reached the restored Notepad");
// The Calculator: in front for a moment when it does not take keys behind (the Store app).
let calc = act(&pc, "run", json!({"program": "calc"}));
let calc = calc["item"]["window"]["id"].as_str().expect("the Calculator's window").to_owned();
std::thread::sleep(Duration::from_secs(1));
let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop);
println!("calculator {calc}; takes keys behind: {}", probe.takes_keys_behind(&calc));
for k in ["escape", "1", "2"] {
let r = act(&pc, "keys", json!({"window": calc, "keys": k}));
assert_eq!(r["ok"], true, "{k}: {r}");
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "{k}: the window in front stays or is back");
}
let shown = text(&act(&pc, "read", json!({"window": calc})));
println!("calculator after 1 2: {}", shown.chars().take(300).collect::<String>());
assert!(shown.contains("12"), "the Calculator shows 12");
let _ = std::fs::remove_dir_all(&data);
}

View File

@@ -226,6 +226,11 @@ LLM과 STT를 함께 켠 측정(M0 `gamemode_stt`, 게임 없음). STT를 먼저
- 프로그램 조작은 [agent-desktop](https://github.com/vercel-labs/agent-desktop)(Apache-2.0, Windows UI Automation)으로 합니다. 앱 설치 파일에 실행 파일(약 4MB)을 함께 넣습니다.
- 창의 버튼·입력칸·메뉴 목록을 글자로 읽고, 고른 항목을 직접 누르거나 값을 넣습니다. 화면 좌표를 쓰지 않으므로 해상도·배율·창 위치가 바뀌어도 맞게 누릅니다.
- 기본은 마우스 커서를 움직이지 않고 포커스도 가져오지 않습니다. 사용자가 다른 창에서 작업하는 중에도 뒤에 있는 창을 조작할 수 있습니다. 최소화된 창은 조작하지 못하므로 먼저 되돌립니다.
- 키 누르기는 섞어 씁니다(2026-11-19 사용자 결정). agent-desktop은 앞에 있는 창에만 키를 보낼 수 있어서, 뒤 창으로 보내는 부분은 앱이 Windows 기능으로 직접 합니다.
- 기본은 뒤 창에 바로 보냅니다. 조합키가 아닌 키 하나(글자, 숫자, Enter, Tab, 방향키 등)를 그 창의 글 입력칸에 보내고, 사용자 화면과 포커스는 그대로 둡니다. Windows는 뒤 창의 어느 칸에 포커스가 있는지 알려 주지 않으므로, 글 입력칸이 하나뿐인 창(메모장 등)에만 이렇게 보냅니다.
- 조합키(Ctrl·Alt·Shift·Win)이거나, 글 입력칸이 하나뿐인 창이 아닐 때(계산기, 스토어·WinUI 앱, 크롬 계열, 파이어폭스 등)만 그 창을 잠깐 앞으로 가져와 키를 누르고, 원래 앞에 있던 창으로 바로 되돌립니다. 그 순간 사용자가 치던 키가 그 창으로 들어갈 수 있습니다.
- 게임 중이거나 전체 화면일 때는 앞으로 가져오지 않고 못 한다고 답합니다.
- 처음부터 키와 프로그램을 보고 한 가지 방식만 고릅니다. 한 방식이 안 됐다고 다른 방식으로 다시 누르지 않으므로 같은 키가 두 번 눌리지 않습니다. 결과는 창을 다시 읽어 확인합니다.
- 누르거나 넣은 뒤 화면을 다시 읽어 결과를 확인하고, 확인된 것만 "했습니다"라고 말합니다.
- Alt+F4, Win+R, Ctrl+Shift+Esc 같은 위험 단축키는 agent-desktop이 기본으로 막습니다. 이 막음을 푸는 옵션(`--force`)은 쓰지 않습니다.
- 관리자 권한으로 실행된 창은 Windows가 막아서 조작하지 못합니다. 그때는 못 한다고 답합니다.

View File

@@ -65,7 +65,7 @@ One tool, `action` picks the operation (read as its first known word):
| `read` | `window` | the window's interactive elements: ref, role, name, value, state | view |
| `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click |
| `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type |
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type |
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`): behind it when it takes them, else in front for a moment (`admin.spec.md`, "PC control") | type |
| `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run |
| `close` | `window` | closes the program's window | click |