feat(v2): pcControl keys go behind a window when they can, else in front for a moment

User decision 2026-11-19 (mixed way). agent-desktop only presses keys in the
window in front, so the app now picks one way before any key is sent:

- behind: one key without modifiers that types or moves is posted to the
  window's one shown text control (Edit, RichEdit, RichEditD2DPT, Scintilla);
  the person's screen and focus stay. Windows does not tell a background
  thread's focused control (win-ci VM: GetFocus is null even attached), so
  only windows with exactly one text control get keys behind.
- in front for a moment: combinations, other keys and every other window;
  the window that was in front is brought back after (also after a failed
  press). A window already in front gets the keys there.
- never in front while game mode is on or the window in front is full screen.
- never both ways, so a key is never pressed twice.

Specs: admin.spec.md "PC control" (Keys), pc-features.md, pc_control.spec.md.
Tests: six new fake-desktop tests in pc.rs; tests/pc_keys_windows.rs on a real
desktop (win-ci VM: letters behind Notepad with Character Map kept in front,
ctrl+a in front and back, the classic Calculator in front and back).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-19 10:41:05 +09:00
parent 999c432216
commit 378ccfafad
5 changed files with 509 additions and 15 deletions

View File

@@ -65,7 +65,7 @@ One tool, `action` picks the operation (read as its first known word):
| `read` | `window` | the window's interactive elements: ref, role, name, value, state | view |
| `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click |
| `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type |
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type |
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`): behind it when it takes them, else in front for a moment (`admin.spec.md`, "PC control") | type |
| `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run |
| `close` | `window` | closes the program's window | click |