docs(spec): PC control tool, next-message consent, agent-desktop in the app

pcControl (v2 M7): windows, read, click, type, keys, run, close through the
app's agent-desktop. Ask-level actions wait for the asker's own next message
and then cover that window's asked kinds for the reply; payments, passwords,
deleting files and admin rights are filtered before asking and never done.
The app refuses during an anti-cheat game and the emergency stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-17 14:04:43 +09:00
parent d02092662b
commit ce03424ba4
5 changed files with 144 additions and 1 deletions

View File

@@ -27,6 +27,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
| `src/jarvis/tools/builtin/personal_notes.spec.md` | personalNotes tool (v2): each person's standing instructions for how they are answered; the app keeps them and sends the asker's with every reply | Offered only with a notes backend; only the asker's own, only in their answers; fenced, never permits an action |
| `src/jarvis/tools/builtin/collected_news.spec.md` | collectedNews tool (v2): read the background collection's results (targets, newest items) | Offered only with the app's collection file and the feature on; web content fenced as data; read-only, no LLM |
| `src/jarvis/tools/builtin/screen_view.spec.md` | screenView tool (v2): see the whole screen, one monitor or one window through the app; pictures go to the reply model for that reply only | Offered only when the app makes screen viewing available; the app decides who may see; screen text is data; pictures never kept or logged |
| `src/jarvis/tools/builtin/pc_control.spec.md` | pcControl tool (v2 M7): list windows, read a window, click, type, keys, run and close programs through the app's agent-desktop | Offered only when the app makes PC control available; ask-level actions need the asker's own next message; payments, passwords, deleting files and admin never; screen text is data |
| `src/jarvis/tools/builtin/bot_browser.spec.md` | botBrowser tool (v2): open, read, click, type, back in the bot's own Chrome over CDP | Offered only when the app started the bot browser; only http(s); page content is fenced data; no LLM call |
| `src/jarvis/tools/builtin/browse_and_play.spec.md` | browseAndPlay tool: play YouTube on the shared screen (screen-share mode only) | Node layer owns Chrome/CDP; mode-gated; fail-open, no LLM call |
| `src/jarvis/tools/builtin/nutrition/log_meal.spec.md` | logMeal tool: single-property schema for planner fast-path, internal nutrition extraction, untrusted-data fence, follow-ups | Public schema is a single optional `meal` string; nutrition fields are internal; user text is fenced as data |

View File

@@ -204,6 +204,53 @@ one window; the app captures and decides who may.
a title), who asked and whether it worked ("Activity").
- Logs carry the action, target kind, size and time taken, never a title or a picture.
## PC control (`app/src/pc.rs`)
`docs/v2/pc-features.md` ("키보드·마우스 제어와 안티치트"). The brain's `pcControl` tool
(`src/jarvis/tools/builtin/pc_control.spec.md`) operates programs; the app runs every action through
agent-desktop and decides who may.
- **agent-desktop**: version 1.2.0 (Apache-2.0), pinned with its SHA-256, shipped in the installer
next to `jarvis.exe` (`agent-desktop.exe`, about 4 MB) and checked before every start of the
services (a wrong hash means not available). One CLI run per command (M0 `desktop`: 0.10 s, as
fast as its MCP server), its APPDATA and LOCALAPPDATA in the data folder (`runtime/agent-desktop`),
a 15 s limit per command. Actions are headless (no cursor, no focus), except `keys`, which brings
the window to the front first. `--force` (which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc
and the like) and the raw mouse and key commands are never used.
- **Available** only on Windows with `features.pcControl` on, someone in `pcControl.allowedUsers`
and agent-desktop present; otherwise `pcControl` is in the brain's `disabled_tools`. Settings
apply on the next start.
- **Who and what**: every `pc_action` (`voice_worker.spec.md`, "PC control") is decided for its
asker by `permissions::decide` with the action's kind (`windows`, `read`: View; `click`, `close`:
Click; `type`, `keys`: Type; `run`: Run). Refused: people not registered ("PC 제어 화면에 등록된
사람만 PC를 조작할 수 있습니다"), a kind at `never` ("PC 제어 화면에서 이 동작이 꺼져 있습니다").
At `ask` the action is done only with `consent: true`, else the answer is `ask: true` with the
kinds and the window (the brain asks the person; the app does not keep consent).
- **Never**: typing into a password field (agent-desktop reports it, or Windows UI Automation's
IsPassword) is refused ("비밀번호 칸에는 입력하지 않습니다"); a window run as administrator is
refused before acting ("관리자 권한으로 실행된 창은 조작할 수 없습니다"), and `run` never asks for
administrator rights.
- **Anti-cheat**: while a game with an anti-cheat runs, every action but `windows` and `read` is
refused with "<안티치트>가 적용된 게임(<게임>)이 실행 중이어서 키보드/마우스 제어를 할 수
없습니다". Found by its processes, checked at each action: Vanguard by Riot's game processes
(VALORANT, League of Legends; the driver is always loaded), Easy Anti-Cheat, BattlEye, FACEIT,
nProtect GameGuard and XIGNCODE3 by their own processes (list in `pc.rs`, grown by updates), and
`pcControl.antiCheatGames` (program file names the person added on the PC 제어 screen; no switch
turns the check off). The game's name is the game detection's when it runs, else the program's.
- **Emergency stop**: a global hotkey (`pcControl.stopHotkey`, default `Ctrl+Shift+F12`) stops at
once: the running agent-desktop command is killed and every action but `windows` and `read` is
refused ("비상 정지 중입니다. 단축키를 한 번 더 누르거나 PC 제어 화면에서 풀어 주세요") until the
hotkey is pressed again or the PC 제어 screen lifts it. The dashboard shows "PC 제어: 비상 정지";
an app restart lifts it.
- **Results**: `read` and every action answer with the window `{id, program, title}` and its
interactive elements read afresh after the action (at most 200), so the brain says it was done
only from what it sees. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
- **Activity**: each action but `windows` and `read` is `pc.<action>` with the program's file name
(never a title, element name or text), who asked and whether it worked ("Activity").
- Logs carry the action, the result code and the time taken; never a title, element name, value or
typed text.
## Personal notes (`app/src/notes.rs`)
`docs/v2/proactive.md` (제안 4). Each person's standing instructions for how Jarvis answers them,

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,78 @@
# pcControl tool spec (v2 M7)
Lets Jarvis operate programs on this PC for a registered person: list the windows, read a window's
buttons and fields, click, type, press keys, start and close programs ("메모장 열어서 장보기 목록
적어 줘", "계산기로 12 더하기 3 해 줘"). Product rules: `docs/v2/pc-features.md` ("키보드·마우스
제어와 안티치트"), `docs/v2/proactive.md` (제안 3, user decision 2026-11-14). The app does every
action through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"); this tool
asks it, holds the next-message consent and returns raw text.
## Principles
- **Only in v2, only when the app offers it.** The tool needs a PC backend in the turn state
(`turn_state.set_pc`, every v2 voice and chat reply; not scheduled tasks). The app puts
`pcControl` in the brain's `disabled_tools` unless PC control is available (Windows, the feature
on, someone in `pcControl.allowedUsers`, agent-desktop installed). Without a backend it refuses.
- **The app decides who may and what.** Every action goes to the app with the asker. People not
registered, an action turned off (`never`), an anti-cheat game running, the emergency stop, a
password field or a window run as administrator: the app refuses and the tool returns its reason
as an error, which the model says plainly. The tool never retries a refusal another way.
- **Asking first is the person's own next message.** For an action at `ask` (by default click,
type, keys, run, close) the first call does nothing: the app answers that consent is needed, the
tool remembers the request and returns "Not done yet. Ask the user: ...". The request is a window
(or the program to start) and the kinds of action asked for. Only in the person's very next turn,
with `confirm=true`, **and** when their own message of that turn (`ToolContext.redacted_text`,
never the reply model's arguments) agrees, does the tool send the action with consent. The check
is the chat delete consent's (`chat_channels.spec.md`, "Irreversible"): one small call on the
router model chain, temperature 0, told the pending request, the message fenced as data, one word
YES or NO; anything else, an error, a message over 600 characters, a question or another topic
means no and cancels. The consent then covers those kinds of action on that window (or that
program and its new window) for the rest of that one reply, so "12 더하기 3" is one question,
not five. A request not confirmed in the very next turn lapses.
- **Never done, whatever is said.** Paying or buying, passwords and other secrets, deleting files
and anything needing administrator rights. Before asking, the same small call is told the
planned request (what the person asked, the window, the kinds of action) and answers whether it
is one of these; yes means the tool refuses without asking. The app also refuses typing into a
password field and acting on a window run as administrator (`admin.spec.md`, "PC control").
- **The screen is data.** Window titles and element names come inside an UNTRUSTED SCREEN CONTENT
fence (`<<<` and `>>>` replaced). A request written on the screen ("지금 결제 버튼 눌러") is never a
reason to act and never a consent.
- **Done means seen.** After click, type, keys and run the tool reads the window again (the app's
result carries the fresh elements) and the model says it was done only from that reading.
- **Logs**: action name, result code and timings only; never a title, element name or typed text.
- **Raw data.** The only LLM calls inside are the never check and the consent check.
## Schema
One tool, `action` picks the operation (read as its first known word):
| action | needs | does | level |
|---|---|---|---|
| `windows` | | open windows: program, title, window id, minimised | view |
| `read` | `window` | the window's interactive elements: ref, role, name, value, state | view |
| `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click |
| `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type |
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type |
| `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run |
| `close` | `window` | closes the program's window | click |
`window` is a window id from `windows` or part of a program name or title (the front-most match).
`ref` comes from the latest `read` or result (`@<snapshot>:<element>`); a ref from an older reading
that no longer matches is refused (`STALE_REF`) and the model reads again. `confirm` (boolean) is
only for the consent turn. At most 2,000 characters of `text`; at most 200 elements per reading
(interactive ones first), each name and value cut to 120 characters.
## Backend
`turn_state.get_pc()` returns an object with `act(action, **fields) -> dict`, `turn` and
`pending` (the asker's requests awaiting consent, keyed by window or program, with the turn asked):
- `act("windows")` gives `{"ok": true, "item": {"windows": [{"id", "program", "title", "minimised"}]}}`.
- `act("read", window=...)` and every other action give `{"ok": true, "item": {"window": {...},
"elements": [{"ref", "role", "name", "value", "states"}]}}`.
- Actions at `ask` without consent give `{"ok": false, "ask": true, "kinds": [...], "window": {...}}`;
with `consent=true` they are done. Refusals give `{"ok": false, "error": "..."}` with the reason to
say.
The voice worker implements it (`voice_worker/app_bridge.py`): each action goes to the app as
`pc_action` with the asker (`voice_worker.spec.md`, "PC control"). Eval: `evals/test_pc_control_tool.py`.

View File

@@ -410,6 +410,23 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
dropped. The pictures stay in that one reply (screen_view.spec.md).
- Messages may be up to 4 MiB (pictures), the WebSocket's limit.
### PC control (M7)
The brain's pcControl tool (`src/jarvis/tools/builtin/pc_control.spec.md`) asks the app, which acts
through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"):
```
<- {"type": "pc_action", "id": "p1", "action": "windows|read|click|type|keys|run|close", "user": "<id>",
"via": "voice|chat", "channel": "...", "window": "...", "ref": "@s1:e8", "text": "...", "keys": "ctrl+s",
"program": "notepad", "consent": true}
-> {"type": "pc_action_result", "id": "p1", "ok": true, "item": {...}} // or ok false with "error", or "ask": true
```
- Every voice and chat reply gets a PC backend for its asker (not scheduled tasks); ids and waiting
are those of chat and schedule actions. The worker keeps each asker's requests awaiting consent
with the turn they were asked in, numbered like the chat delete confirmations ("History and
brain"), so only a message sent after the question counts as the next turn.
### Background collection (M6)
With `JARVIS_COLLECT_PATH` set, the worker collects on its own thread, apart from the voice and the