feat: screen viewing, Jarvis sees the whole screen, a monitor or a window for registered people

- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
  title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
  viewing"), decides for the asker (only people registered for PC control), never picks, lists
  or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
  minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
  OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
  not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
  game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-15 17:02:59 +09:00
parent bd7b69d4f4
commit 53bccb620d
39 changed files with 1844 additions and 68 deletions

View File

@@ -65,10 +65,10 @@ manages the MCP servers it connects.
`browserShare` (봇용 크롬창 화면공유 선택 표시, the bot browser on the virtual monitor, below
"Virtual monitor"), off by default; `schedule` (reminders and scheduled tasks, below
"Schedules", and the brain tool `schedule`), on by default; `notes` (personal notes, below
"Personal notes", and the brain tool `personalNotes`), on by default. The tools of a feature that
"Personal notes", and the brain tool `personalNotes`), on by default; `screen` (화면 보기, below
"Screen viewing", and the brain tool `screenView`), on by default. The tools of a feature that
is off go into the brain's `disabled_tools`: they are neither offered nor run
(`voice_worker.spec.md`, "Tools"). The other features of the 기능 screen (PC control, screen
viewing) come with their milestone (M7).
(`voice_worker.spec.md`, "Tools"). PC control comes with its milestone (M7).
- `mcpServers` (new list): each `{name, command, args, env, enabled}`. `name` is 1-32 letters,
digits, `-` or `_` without `__` (the brain names a server's tools `<name>__<tool>`), not listed
twice; `command` is not empty; `args` a list of strings; `env` the names of the environment
@@ -162,6 +162,42 @@ decision 2026-11-14). What Jarvis may do on this PC and for whom; the PC control
- The PC 제어 screen edits the section and shows, read-only, what is never done and each registered
chat channel's send and delete switches (changed on the 채팅 채널 screen).
## Screen viewing (`app/src/screen.rs`)
`docs/v2/pc-features.md` ("화면 보기", "화면 질문을 빠르게"). The brain's `screenView` tool
(`src/jarvis/tools/builtin/screen_view.spec.md`) sees all monitors as one picture, one monitor, or
one window; the app captures and decides who may.
- **Available** only on Windows with `features.screen` on, someone in `pcControl.allowedUsers` and
`pcControl.view` not `never`. Otherwise `screenView` is in the brain's `disabled_tools` and the
LLM reads no pictures (`--vision off`, below "LLM"). Settings apply on the next start.
- **Who**: every `screen_action` (`voice_worker.spec.md`, "Screen viewing") is decided for its
asker by `permissions::decide(View)`: people not registered are refused ("PC 제어 화면에 등록된
사람만 화면을 볼 수 있습니다"), and so is viewing at `ask` until the PC control tool brings the
next-message consent (M7, next step). Viewing works while an anti-cheat game runs (it only reads).
- **Capture**: Windows Graphics Capture (D3D11, no cursor, no yellow border where Windows allows
it), one session per request, the first frame within 2 s. Monitors are numbered as Windows
numbers them (`\\.\DISPLAY<n>`); the bot browser's virtual monitor ("Virtual monitor") is left out
of the list and of the whole screen. `screen` puts all monitors in one picture as they are placed;
`window` takes the visible top-level window whose title or program name contains the text
(case-insensitive; the one nearest the front when several do), also when others cover it; a
minimised window is refused ("최소화되어 있어 볼 수 없습니다").
- **Exclusions**: `pcControl.screenExclude` (program file names, ".exe" may be left out; default
`1Password`, `Bitwarden`, `KeePass`, `KeePassXC`). Their windows are never captured or listed, and
in a monitor or whole-screen picture their area is painted black before anything leaves the
capture.
- **Pictures**: scaled down (area average) so the long side is at most 768 px (`fast`), 1280 px
(`normal`) or, for `detailed`, cut in four parts of at most 1280 px each (a picture whose long side
is at most 1600 px stays one part at full size); JPEG quality 80. At most 3 MiB of pictures per
answer, else refused. A picture that is all black is refused ("화면이 검게 나와 볼 수 없습니다",
protected video, exclusive full screen). Pictures go only into the `screen_action_result`: never
saved, never logged.
- **List**: monitors `{number, width, height, main}` and visible windows `{program, title,
minimised}`, front first, at most 30, without excluded programs and the app's own windows.
- **Activity**: each capture is `screen.view` with the target (`전체 화면`, `모니터 <n>`, `창`; never
a title), who asked and whether it worked ("Activity").
- Logs carry the action, target kind, size and time taken, never a title or a picture.
## Personal notes (`app/src/notes.rs`)
`docs/v2/proactive.md` (제안 4). Each person's standing instructions for how Jarvis answers them,
@@ -184,7 +220,7 @@ editable on the 사람별 지침 screen.
`docs/v2/proactive.md` (제안 5). What Jarvis did outside its answers, for the 한 일 screen: each chat
channel action the brain's tool asked for (`chat.send`, `chat.edit`, `chat.react`, `chat.delete`,
`chat.pin`, `chat.unpin`), each schedule delivery (`schedule.remind`, `schedule.task`) and each
news notification (`collect.notify`), done or failed. PC control actions join it with M7. Ordinary answers are not recorded.
news notification (`collect.notify`), each screen capture (`screen.view`, "Screen viewing"), done or failed. The other PC control actions join it with M7. Ordinary answers are not recorded.
- One JSON line per action in `activity/<YYYY-MM-DD>.jsonl` (the folder is made at the first one):
`{at, kind, target, by, ok, note}`. `target` is the channel (and message) id or `예약 <id>`; `by`
@@ -530,14 +566,15 @@ The brain answers with one model, Qwen3.6-35B-A3B UD-Q4_K_XL through Strata (M0)
start it first, so nothing answers before the brain can, and stop it last.
- **Strata from the data folder** (the default): `strata launch --family qwen36 --model ud-q4_k_xl
--data-dir models/llm --context <llm.context> --vision off --parallel <n> --access local --port
--data-dir models/llm --context <llm.context> --vision <off|gpu|cpu> --parallel <n> --access local --port
<llm.port> --no-api-key --restart --yes --no-tui`, run like an install command (the layout's
environment, `runtime/strata` as the working folder, one process group, lines to the `llm` log
tab). It downloads the model on the first start, starts the engine and returns; the app then
waits (up to 30 minutes) until `/v1/models` lists the model and takes its id. `--restart`
replaces a server left with other choices; the same choices are reused. `--parallel` is 1 (faster
on a 16 GB card, M0) and 2 when a chat channel answers every message, so its answers do not wait
behind voice answers. Pictures are off until something in v2 sends one. Stopping the services
behind voice answers. Pictures (`--vision`) are off unless screen viewing is available ("Screen viewing"): then `gpu`
in the normal stage and `cpu` in the game stage (M0). Stopping the services
runs `strata launch --stop` (ended after 2 minutes, with everything it started) and then checks
that the port refuses connections; so does a failed or cancelled start, so nothing half-started
keeps graphics memory (each check waits up to 10 s to connect: Windows refuses a closed local
@@ -559,7 +596,8 @@ start it first, so nothing answers before the brain can, and stop it last.
left. What the model holds without experts is measured once per context, number of requests at
once and card (its NVIDIA UUID): the first-run self-check, a start with `expert-vram=0`, one
short answer, then the card's memory again and the staging from the engine log; kept in
`config/llm-probe.json` under `<context>/<parallel>`, and the real start then replaces that
`config/llm-probe.json` under `<context>/<parallel>` (`<context>/<parallel>/vision` with pictures
on the card), and the real start then replaces that
server. A measurement whose answer or second reading fails (or shows nothing more held) is not
kept: the model starts with `expert-vram=0` and the next start measures again. Before the card
is read, `strata launch --stop` always runs and the start's port must refuse connections, so a
@@ -821,7 +859,15 @@ also with the user idle past Windows' focus lock, no taskbar or Alt+Tab entry fo
left after the stop, and the uninstall leaving no device, driver package, registry key or settings
folder. Game detection (`tests/game_watch.rs`): a program started from a Steam game folder (also under
a path with non-ASCII letters) or from a Battle.net game's install folder (a test uninstall entry)
turns game mode on with its name and off once it ends. Performance stages (on every system, with a fake
turns game mode on with its name and off once it ends. Screen viewing (`tests/screen_windows.rs`,
`JARVIS_TEST_SCREEN=1`): the monitors and Notepad's window are listed, the window, each monitor and
the whole screen come back as JPEG pictures of their size, and with Notepad excluded it is neither
picked nor listed and its area is black on the whole screen; someone not registered is refused. On
every system with a fake screen: refusals before any capture, monitors by number, the whole screen
as placed with excluded areas black (also at another display scale), windows by title or program
front first (never excluded, secondary or the app's own), minimised and black pictures refused, the
detail sizes and parts; the worker protocol test carries a 4K capture's pictures to a fake brain and
back. Performance stages (on every system, with a fake
Strata and card): a game moves the LLM to the game placement (measured once at 16K and one request,
2 GiB on the game PC's numbers, half the 16 engine threads) and back after the wait, pause stops
it, a card with no room for a game pauses, the normal stage counts a program opened since, the game