Compare commits
5 Commits
app-v0.1.2
...
app-v0.1.2
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
61b68130c6 | ||
|
|
f1600f09c2 | ||
|
|
ce03424ba4 | ||
|
|
d02092662b | ||
|
|
4adc6a8d72 |
@@ -27,6 +27,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
|
||||
| `src/jarvis/tools/builtin/personal_notes.spec.md` | personalNotes tool (v2): each person's standing instructions for how they are answered; the app keeps them and sends the asker's with every reply | Offered only with a notes backend; only the asker's own, only in their answers; fenced, never permits an action |
|
||||
| `src/jarvis/tools/builtin/collected_news.spec.md` | collectedNews tool (v2): read the background collection's results (targets, newest items) | Offered only with the app's collection file and the feature on; web content fenced as data; read-only, no LLM |
|
||||
| `src/jarvis/tools/builtin/screen_view.spec.md` | screenView tool (v2): see the whole screen, one monitor or one window through the app; pictures go to the reply model for that reply only | Offered only when the app makes screen viewing available; the app decides who may see; screen text is data; pictures never kept or logged |
|
||||
| `src/jarvis/tools/builtin/pc_control.spec.md` | pcControl tool (v2 M7): list windows, read a window, click, type, keys, run and close programs through the app's agent-desktop | Offered only when the app makes PC control available; ask-level actions need the asker's own next message; payments, passwords, deleting files and admin never; screen text is data |
|
||||
| `src/jarvis/tools/builtin/bot_browser.spec.md` | botBrowser tool (v2): open, read, click, type, back in the bot's own Chrome over CDP | Offered only when the app started the bot browser; only http(s); page content is fenced data; no LLM call |
|
||||
| `src/jarvis/tools/builtin/browse_and_play.spec.md` | browseAndPlay tool: play YouTube on the shared screen (screen-share mode only) | Node layer owns Chrome/CDP; mode-gated; fail-open, no LLM call |
|
||||
| `src/jarvis/tools/builtin/nutrition/log_meal.spec.md` | logMeal tool: single-property schema for planner fast-path, internal nutrition extraction, untrusted-data fence, follow-ups | Public schema is a single optional `meal` string; nutrition fields are internal; user text is fenced as data |
|
||||
|
||||
2
app/Cargo.lock
generated
2
app/Cargo.lock
generated
@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.22"
|
||||
version = "0.1.24"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.22"
|
||||
version = "0.1.24"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"
|
||||
|
||||
4
app/shell/Cargo.lock
generated
4
app/shell/Cargo.lock
generated
@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.22"
|
||||
version = "0.1.24"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
@@ -2666,7 +2666,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.22"
|
||||
version = "0.1.24"
|
||||
dependencies = [
|
||||
"jarvis-app",
|
||||
"serde_json",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.22"
|
||||
version = "0.1.24"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"
|
||||
|
||||
@@ -204,6 +204,53 @@ one window; the app captures and decides who may.
|
||||
a title), who asked and whether it worked ("Activity").
|
||||
- Logs carry the action, target kind, size and time taken, never a title or a picture.
|
||||
|
||||
## PC control (`app/src/pc.rs`)
|
||||
|
||||
`docs/v2/pc-features.md` ("키보드·마우스 제어와 안티치트"). The brain's `pcControl` tool
|
||||
(`src/jarvis/tools/builtin/pc_control.spec.md`) operates programs; the app runs every action through
|
||||
agent-desktop and decides who may.
|
||||
|
||||
- **agent-desktop**: version 1.2.0 (Apache-2.0), pinned with its SHA-256, shipped in the installer
|
||||
next to `jarvis.exe` (`agent-desktop.exe`, about 4 MB) and checked before every start of the
|
||||
services (a wrong hash means not available). One CLI run per command (M0 `desktop`: 0.10 s, as
|
||||
fast as its MCP server), its APPDATA and LOCALAPPDATA in the data folder (`runtime/agent-desktop`),
|
||||
a 15 s limit per command. Actions are headless (no cursor, no focus), except `keys`, which brings
|
||||
the window to the front first. `--force` (which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc
|
||||
and the like) and the raw mouse and key commands are never used.
|
||||
- **Available** only on Windows with `features.pcControl` on, someone in `pcControl.allowedUsers`
|
||||
and agent-desktop present; otherwise `pcControl` is in the brain's `disabled_tools`. Settings
|
||||
apply on the next start.
|
||||
- **Who and what**: every `pc_action` (`voice_worker.spec.md`, "PC control") is decided for its
|
||||
asker by `permissions::decide` with the action's kind (`windows`, `read`: View; `click`, `close`:
|
||||
Click; `type`, `keys`: Type; `run`: Run). Refused: people not registered ("PC 제어 화면에 등록된
|
||||
사람만 PC를 조작할 수 있습니다"), a kind at `never` ("PC 제어 화면에서 이 동작이 꺼져 있습니다").
|
||||
At `ask` the action is done only with `consent: true`, else the answer is `ask: true` with the
|
||||
kinds and the window (the brain asks the person; the app does not keep consent).
|
||||
- **Never**: typing into a password field (agent-desktop reports it, or Windows UI Automation's
|
||||
IsPassword) is refused ("비밀번호 칸에는 입력하지 않습니다"); a window run as administrator is
|
||||
refused before acting ("관리자 권한으로 실행된 창은 조작할 수 없습니다"), and `run` never asks for
|
||||
administrator rights.
|
||||
- **Anti-cheat**: while a game with an anti-cheat runs, every action but `windows` and `read` is
|
||||
refused with "<안티치트>가 적용된 게임(<게임>)이 실행 중이어서 키보드/마우스 제어를 할 수
|
||||
없습니다". Found by its processes, checked at each action: Vanguard by Riot's game processes
|
||||
(VALORANT, League of Legends; the driver is always loaded), Easy Anti-Cheat, BattlEye, FACEIT,
|
||||
nProtect GameGuard and XIGNCODE3 by their own processes (list in `pc.rs`, grown by updates), and
|
||||
`pcControl.antiCheatGames` (program file names the person added on the PC 제어 screen; no switch
|
||||
turns the check off). The game's name is the game detection's when it runs, else the program's.
|
||||
- **Emergency stop**: a global hotkey (`pcControl.stopHotkey`, default `Ctrl+Shift+F12`) stops at
|
||||
once: the running agent-desktop command is killed and every action but `windows` and `read` is
|
||||
refused ("비상 정지 중입니다. 단축키를 한 번 더 누르거나 PC 제어 화면에서 풀어 주세요") until the
|
||||
hotkey is pressed again or the PC 제어 screen lifts it. The dashboard shows "PC 제어: 비상 정지";
|
||||
an app restart lifts it.
|
||||
- **Results**: `read` and every action answer with the window `{id, program, title}` and its
|
||||
interactive elements read afresh after the action (at most 200), so the brain says it was done
|
||||
only from what it sees. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
|
||||
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
|
||||
- **Activity**: each action but `windows` and `read` is `pc.<action>` with the program's file name
|
||||
(never a title, element name or text), who asked and whether it worked ("Activity").
|
||||
- Logs carry the action, the result code and the time taken; never a title, element name, value or
|
||||
typed text.
|
||||
|
||||
## Personal notes (`app/src/notes.rs`)
|
||||
|
||||
`docs/v2/proactive.md` (제안 4). Each person's standing instructions for how Jarvis answers them,
|
||||
@@ -582,8 +629,8 @@ between the normal placement and the game placement as games come and go (not wi
|
||||
engine's CPU threads (read from its log at the normal start, else a quarter of the logical CPUs),
|
||||
and the game placement (`placement.rs`: the reserve beside more than 2 GiB of experts, else
|
||||
2 GiB, else 0 GiB, else pause), measured once at that context like the normal one. Other
|
||||
programs' memory is what the card holds after the LLM stopped, less the STT worker's model while
|
||||
it is loaded (1300 MiB), read afresh at every placement. A normal placement made with no game
|
||||
programs' memory is what the card holds after the LLM stopped, less the voice worker's models
|
||||
while they are loaded (the STT model 1300 MiB, the TTS 1005 MiB), read afresh at every placement. A normal placement made with no game
|
||||
running keeps its reading; one made while a game runs (the way back from a failed game stage)
|
||||
keeps the earlier one. The game stage counts other programs as that kept reading and leaves
|
||||
a reserve of at least what has grown since (the game and any program opened after that
|
||||
@@ -591,7 +638,7 @@ between the normal placement and the game placement as games come and go (not wi
|
||||
as the services started, there is no such reading: game detection looks once before the LLM
|
||||
starts, at the window in front and at every running program (one in a launcher's game folder or
|
||||
the person's games list counts, also behind other windows; how a window looks is not known
|
||||
there). Then the game stage counts what the card holds now, the game included, and leaves only
|
||||
there; the `game` log tab gets "a game runs as the services start: program"). Then the game stage counts what the card holds now, the game included, and leaves only
|
||||
the room for other programs. The dashboard shows the stage that runs, also on a fallback
|
||||
placement without experts.
|
||||
- Pause: the LLM stops (`llm` off, "게임 중 일시정지"); the brain answers nothing until the stage
|
||||
@@ -599,7 +646,10 @@ between the normal placement and the game placement as games come and go (not wi
|
||||
- The foreground program is a game, in this order: in `notGames`: no; in `games`: yes
|
||||
(`registered`); a browser (also the bot's Chrome), video player, the desktop, Discord, a
|
||||
launcher's own window or this app: no; under a launcher's game folder (the folder after
|
||||
`steamapps\common`, `Epic Games` or `Riot Games`) or inside a Battle.net game's install folder:
|
||||
`steamapps\common`, `Epic Games` or `Riot Games`), unless that folder is the launcher's own or a
|
||||
program that is not a game (`Steamworks Shared` and `wallpaper_engine` (Wallpaper Engine, which
|
||||
runs all the time) under Steam, `Launcher` and `Epic Online Services` under Epic, `Riot Client`
|
||||
under Riot), or inside a Battle.net game's install folder:
|
||||
yes (`launcher`, with the launcher's name). Battle.net installs anywhere, so its games come from
|
||||
the machine's uninstall entries (64- and 32-bit), read when the watch starts: entries removed by
|
||||
its "Blizzard Uninstaller" or published by Blizzard Entertainment, with an install folder, not
|
||||
@@ -644,7 +694,8 @@ start it first, so nothing answers before the brain can, and stop it last.
|
||||
and leaves the speech-to-text worker no room, and its qwen36 engine takes no reserve. Experts get
|
||||
what is left of the card after what it holds before the start (other programs), the model without
|
||||
experts, the STT worker's share (2740 MiB: its model and ten speakers at once, M0 `stt`), the
|
||||
engine's margin (768 MiB) and the room for programs opened later (`game.otherRoomGb`, default
|
||||
TTS's share (1005 MiB, MeloTTS after its warm-up sentence, M0 `tts_melo`; it loads after the LLM,
|
||||
so a start and a switch with the same programs open place the same), the engine's margin (768 MiB) and the room for programs opened later (`game.otherRoomGb`, default
|
||||
1 GB); the budget first holds the long-prompt staging, rounded down to 0.1 GiB, 0 when nothing is
|
||||
left. What the model holds without experts is measured once per context, number of requests at
|
||||
once and card (its NVIDIA UUID): the first-run self-check, a start with `expert-vram=0`, one
|
||||
@@ -658,7 +709,7 @@ start it first, so nothing answers before the brain can, and stop it last.
|
||||
counted as other programs' memory; when that stop is not confirmed, nothing is read or kept and
|
||||
the model starts with `expert-vram=0`. Without a card nothing is set. The game
|
||||
placement (used by the switching that comes next) takes the reserve when it fits beside more
|
||||
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from.
|
||||
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from, and once the model answers how much of the card is left free (`card after the start: N MiB free of T MiB`).
|
||||
- **A server the app does not start**: with `llm.url` (an OpenAI-compatible `/v1` address, in the
|
||||
settings file only) the app starts nothing; that server must list a model within 30 seconds.
|
||||
- **Settings**: `llm.context` (default 32768 tokens per request, 4096-262144) and `llm.port`
|
||||
@@ -994,7 +1045,8 @@ back. Performance stages (on every system, with a fake
|
||||
Strata and card): a game moves the LLM to the game placement (measured once at 16K and one request,
|
||||
2 GiB on the game PC's numbers, half the 16 engine threads) and back after the wait, pause stops
|
||||
it, a card with no room for a game pauses, the normal stage counts a program opened since, the game
|
||||
stage does not count the game twice and counts what grew since the normal reading, a game running
|
||||
stage does not count the game twice and counts what grew since the normal reading, the voice
|
||||
worker's models loaded after the start leave the normal placement as it started, a game running
|
||||
as the services start does not pause the LLM, the way back to normal from a failed game stage while
|
||||
the game runs keeps the reading from before the game, a game program behind other windows is found
|
||||
at the start (win-ci VM: a minimised program in a Steam game folder), a fallback game placement still shows as the game
|
||||
@@ -1004,7 +1056,8 @@ is tried three times; after that a new game tries again, and the next try waits
|
||||
failed switch was over. The worker client ignores a late idle of an earlier switch and sends the
|
||||
latest LLM state first when a worker connects again. On every system: game detection's rules
|
||||
(a borderless game, a maximised program, a browser's full-screen video, exclusive full screen,
|
||||
launcher folders and their own windows, Battle.net games by install folder and their uninstall
|
||||
launcher folders, their own windows and own folders (Wallpaper Engine, the Riot client's crash
|
||||
handler), Battle.net games by install folder and their uninstall
|
||||
entries, non-ASCII paths, the person's lists first, staying on until the game ends,
|
||||
detection off) and its settings. A removal forgets the driver only when no device is left and the package
|
||||
is gone or was not in the store; a failed install forgets it only under the same rule (and never
|
||||
|
||||
@@ -28,12 +28,14 @@ const NOT_GAMES: &[&str] = &[
|
||||
];
|
||||
|
||||
/// Launcher game folders, as the path parts before the game's own folder: a program under one
|
||||
/// counts as a game whatever its window looks like. Battle.net installs anywhere, so its games come
|
||||
/// from their install folders (`InstalledGame`).
|
||||
const LAUNCHERS: &[(&[&str], &str)] = &[
|
||||
(&["steamapps", "common"], "Steam"),
|
||||
(&["epic games"], "Epic"),
|
||||
(&["riot games"], "Riot"),
|
||||
/// counts as a game whatever its window looks like, unless that folder is one of the launcher's
|
||||
/// own (its client, shared runtimes, or a non-game app such as Wallpaper Engine, which runs all
|
||||
/// the time). Battle.net installs anywhere, so its games come from their install folders
|
||||
/// (`InstalledGame`).
|
||||
const LAUNCHERS: &[(&[&str], &str, &[&str])] = &[
|
||||
(&["steamapps", "common"], "Steam", &["steamworks shared", "wallpaper_engine"]),
|
||||
(&["epic games"], "Epic", &["launcher", "epic online services"]),
|
||||
(&["riot games"], "Riot", &["riot client"]),
|
||||
];
|
||||
|
||||
/// A game a launcher installed somewhere of its own choosing.
|
||||
@@ -71,10 +73,11 @@ fn launcher_game(path: &str, installed: &[InstalledGame]) -> Option<(&'static st
|
||||
return Some((g.launcher, g.name.clone()));
|
||||
}
|
||||
}
|
||||
for (marker, launcher) in LAUNCHERS {
|
||||
for (marker, launcher, own) in LAUNCHERS {
|
||||
// The game's folder follows the marker, and the program is inside it.
|
||||
if let Some(i) = (0..p.len().saturating_sub(marker.len() + 1)).find(|&i| same(&p[i..i + marker.len()], marker)) {
|
||||
return Some((launcher, p[i + marker.len()].to_owned()));
|
||||
let folder = p[i + marker.len()];
|
||||
return (!own.iter().any(|o| folder.eq_ignore_ascii_case(o))).then(|| (*launcher, folder.to_owned()));
|
||||
}
|
||||
}
|
||||
None
|
||||
@@ -150,14 +153,14 @@ pub fn classify(fg: &Foreground, d3d_fullscreen: bool, s: &GameSettings, install
|
||||
(fg.covers_monitor && fg.no_frame && fg.gpu_3d >= MIN_GPU_3D).then_some(Reason::Borderless)
|
||||
}
|
||||
|
||||
/// Whether one of these programs (full paths) is a game by what it is: the person's list or a
|
||||
/// The first of these programs (full paths) that is a game by what it is: the person's list or a
|
||||
/// launcher's game folder. Used once as the services start, for games behind other windows (how a
|
||||
/// window looks is not known there, so those rules do not apply).
|
||||
pub fn game_running_among(paths: &[String], s: &GameSettings, installed: &[InstalledGame]) -> bool {
|
||||
paths.iter().any(|p| {
|
||||
let fg = Foreground { pid: 0, path: p.clone(), covers_monitor: false, no_frame: false, gpu_3d: 0.0 };
|
||||
pub fn game_running_among(paths: &[String], s: &GameSettings, installed: &[InstalledGame]) -> Option<String> {
|
||||
paths.iter().find(|p| {
|
||||
let fg = Foreground { pid: 0, path: (*p).clone(), covers_monitor: false, no_frame: false, gpu_3d: 0.0 };
|
||||
classify(&fg, false, s, installed).is_some()
|
||||
})
|
||||
}).cloned()
|
||||
}
|
||||
|
||||
/// A game's name for the dashboard: its name from the launcher (the folder under a launcher's game
|
||||
@@ -225,7 +228,11 @@ impl GameWatch {
|
||||
if cfg!(windows) && settings.detect {
|
||||
let snap = probe::Probe::new().snapshot();
|
||||
state.lock().unwrap().update(&snap, &settings, probe::alive);
|
||||
background = game_running_among(&probe::process_paths(), &settings, &snap.installed);
|
||||
if let Some(path) = game_running_among(&probe::process_paths(), &settings, &snap.installed) {
|
||||
// Which program, so a wrong one can be put on the "not games" list.
|
||||
crate::info_log!("game", "a game runs as the services start: {}", file_name(&path));
|
||||
background = true;
|
||||
}
|
||||
}
|
||||
let thread = (cfg!(windows) && settings.detect).then(|| {
|
||||
let (state, stop) = (state.clone(), stop.clone());
|
||||
@@ -622,15 +629,35 @@ mod tests {
|
||||
let s = settings();
|
||||
let wow = InstalledGame { dir: r"D:\Games\World of Warcraft".into(), name: "World of Warcraft".into(), launcher: "Battle.net" };
|
||||
// Behind other windows: only what the program is (where it lives, or the person's list).
|
||||
assert!(game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &s, &[]));
|
||||
assert!(game_running_among(&[r"D:\Games\World of Warcraft\_retail_\Wow.exe".into()], &s, &[wow]));
|
||||
assert!(!game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\Riot Client\RiotClientServices.exe".into(),
|
||||
r"C:\Program Files\Google\Chrome\Application\chrome.exe".into()], &s, &[]), "launchers and browsers are not games");
|
||||
assert_eq!(game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &s, &[]),
|
||||
Some(r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()), "the program found, for the log");
|
||||
assert!(game_running_among(&[r"D:\Games\World of Warcraft\_retail_\Wow.exe".into()], &s, &[wow]).is_some());
|
||||
assert!(game_running_among(&[r"C:\Windows\explorer.exe".into(), r"C:\Riot Games\Riot Client\RiotClientServices.exe".into(),
|
||||
r"C:\Program Files\Google\Chrome\Application\chrome.exe".into()], &s, &[]).is_none(), "launchers and browsers are not games");
|
||||
let mut mine = settings();
|
||||
mine.games = vec!["mygame".into()];
|
||||
assert!(game_running_among(&[r"D:\x\MyGame.exe".into()], &mine, &[]));
|
||||
assert!(game_running_among(&[r"D:\x\MyGame.exe".into()], &mine, &[]).is_some());
|
||||
mine.not_games = vec!["VALORANT.exe".into()];
|
||||
assert!(!game_running_among(&[r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &mine, &[]));
|
||||
assert!(game_running_among(&[r"C:\Riot Games\VALORANT\live\VALORANT.exe".into()], &mine, &[]).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn programs_in_a_launchers_own_folders_are_not_games() {
|
||||
let s = settings();
|
||||
// Always running on the game PC (2026-11-17): Wallpaper Engine is a Steam app, and the Riot
|
||||
// client's crash handler sits in the client's folder.
|
||||
for p in [r"C:\Program Files (x86)\Steam\steamapps\common\wallpaper_engine\wallpaper64.exe",
|
||||
r"C:\Program Files (x86)\Steam\steamapps\common\Steamworks Shared\_CommonRedist\vcredist\x.exe",
|
||||
r"C:\Riot Games\Riot Client\RiotClientCrashHandler.exe",
|
||||
r"C:\Program Files\Epic Games\Launcher\Portal\Binaries\Win64\EpicGamesLauncher.exe",
|
||||
r"C:\Program Files (x86)\Epic Games\Epic Online Services\service\EpicOnlineServices.exe"] {
|
||||
assert_eq!(classify(&fg(p, false, false, 0.0), false, &s, &[]), None, "{p}");
|
||||
}
|
||||
assert_eq!(game_running_among(&[r"C:\Riot Games\Riot Client\RiotClientCrashHandler.exe".into(),
|
||||
r"C:\Program Files (x86)\Steam\steamapps\common\wallpaper_engine\wallpaper64.exe".into()], &s, &[]), None);
|
||||
// The games beside them still are.
|
||||
assert_eq!(classify(&fg(r"C:\Program Files\Epic Games\Fortnite\FortniteClient.exe", false, false, 0.0), false, &s, &[]),
|
||||
Some(Reason::Launcher("Epic")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -13,6 +13,15 @@ pub const MARGIN_MIB: i64 = 768;
|
||||
pub const STT_MIB: i64 = STT_MODEL_MIB + 1440;
|
||||
/// The STT worker's model alone (M0 `stt`).
|
||||
pub const STT_MODEL_MIB: i64 = 1300;
|
||||
/// The TTS (MeloTTS, Korean) on the card after its warm-up sentence, measured on the game PC
|
||||
/// (M0 `tts_melo`). It loads after the LLM starts, so it is kept beside like the STT's share.
|
||||
pub const TTS_MIB: i64 = 1005;
|
||||
|
||||
/// What the voice worker's models hold on the card now: the STT model and the TTS once loaded.
|
||||
/// Taken off what the card holds, so they are not counted again as other programs.
|
||||
pub fn worker_models_mib(stt_loaded: bool, tts_loaded: bool) -> i64 {
|
||||
(if stt_loaded { STT_MODEL_MIB } else { 0 }) + if tts_loaded { TTS_MIB } else { 0 }
|
||||
}
|
||||
/// The smallest expert budget in a game: it holds the long-prompt staging (about 1.1 GiB) first,
|
||||
/// so 2 GiB is the least that loads experts, and it was 1.5-2.1 times faster than none in a real
|
||||
/// game (user decision, 2026-11-10).
|
||||
@@ -36,10 +45,10 @@ pub struct Probe {
|
||||
}
|
||||
|
||||
/// The expert budget (GiB, down to 0.1) that leaves `room_mib` free besides what the card already
|
||||
/// holds (`used_mib`), the LLM without experts, the STT's share and the engine's margin. The budget
|
||||
/// holds (`used_mib`), the LLM without experts, the STT's and TTS's shares and the engine's margin. The budget
|
||||
/// first holds the staging, which the start without experts already has (M0 `expert_budget_gib`).
|
||||
pub fn expert_budget_gib(total_mib: i64, used_mib: i64, p: &Probe, room_mib: i64) -> f64 {
|
||||
let spare = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB - room_mib;
|
||||
let spare = total_mib - used_mib - p.model_mib - STT_MIB - TTS_MIB - MARGIN_MIB - room_mib;
|
||||
if spare <= 0 {
|
||||
return 0.0;
|
||||
}
|
||||
@@ -71,7 +80,7 @@ pub fn game(total_mib: i64, used_mib: i64, p: &Probe, reserve_mib: i64) -> GameP
|
||||
return GamePlacement { expert_gib: expert, game_room_mib: reserve_mib, paused: false };
|
||||
}
|
||||
// Beside the model without experts (which already holds the staging): what 2 GiB adds.
|
||||
let free = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB;
|
||||
let free = total_mib - used_mib - p.model_mib - STT_MIB - TTS_MIB - MARGIN_MIB;
|
||||
let extra = ((MIN_GAME_EXPERT_GIB - p.staging_gib) * 1024.0).max(0.0) as i64;
|
||||
if free - extra > 0 {
|
||||
return GamePlacement { expert_gib: MIN_GAME_EXPERT_GIB, game_room_mib: free - extra, paused: false };
|
||||
@@ -93,7 +102,7 @@ mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The game PC in M0 `gamemode_tune` (RTX 5070 Ti 16 GB): 2760 MiB held before the LLM there
|
||||
/// included the STT model (1300 MiB), which the app starts after the LLM.
|
||||
/// included the STT model (1300 MiB), which the app starts after the LLM. M0 ran no TTS.
|
||||
const TOTAL: i64 = 16303;
|
||||
const OTHERS: i64 = 2760 - 1300;
|
||||
fn probe32() -> Probe {
|
||||
@@ -103,7 +112,8 @@ mod tests {
|
||||
#[test]
|
||||
fn the_normal_placement_is_the_one_m0_measured_with_room_for_the_stt() {
|
||||
// M0 got 1.5 GiB (the STT model counted as already held); the app counts it in the STT share.
|
||||
assert_eq!(normal(TOTAL, OTHERS, &probe32(), 1024), 1.5);
|
||||
// M0 ran no TTS: the app keeps its share too, so the same 1.5 GiB needs that much less held.
|
||||
assert_eq!(normal(TOTAL, OTHERS - TTS_MIB, &probe32(), 1024), 1.5);
|
||||
// More room for other programs, fewer experts; no room at all, none.
|
||||
assert!(normal(TOTAL, OTHERS, &probe32(), 2048) < 1.5);
|
||||
assert_eq!(normal(TOTAL, OTHERS + 8000, &probe32(), 1024), 0.0);
|
||||
@@ -116,7 +126,7 @@ mod tests {
|
||||
// 8 GB does not fit on 16 GB: the smallest placement, and the game gets what is left.
|
||||
let g = game(TOTAL, OTHERS, &probe16, 8192);
|
||||
assert_eq!((g.expert_gib, g.paused), (MIN_GAME_EXPERT_GIB, false));
|
||||
let expected = TOTAL - OTHERS - probe16.model_mib - STT_MIB - MARGIN_MIB - ((2.0 - 1.1) * 1024.0) as i64;
|
||||
let expected = TOTAL - OTHERS - probe16.model_mib - STT_MIB - TTS_MIB - MARGIN_MIB - ((2.0 - 1.1) * 1024.0) as i64;
|
||||
assert_eq!(g.game_room_mib, expected);
|
||||
assert!(g.game_room_mib > 0 && g.game_room_mib < 8192);
|
||||
// A bigger card keeps the whole reserve, with more experts than the minimum.
|
||||
@@ -124,9 +134,9 @@ mod tests {
|
||||
assert_eq!(big.game_room_mib, 8192);
|
||||
assert!(big.expert_gib > MIN_GAME_EXPERT_GIB && !big.paused);
|
||||
// No room beside 2 GiB but some beside none: the 0 GiB placement, before pausing.
|
||||
let tight = game(TOTAL, 2960, &Probe { model_mib: 9124, ..probe16.clone() }, 8192);
|
||||
let tight = game(TOTAL, 1955, &Probe { model_mib: 9124, ..probe16.clone() }, 8192);
|
||||
assert_eq!((tight.expert_gib, tight.paused), (0.0, false));
|
||||
assert_eq!(tight.game_room_mib, TOTAL - 2960 - 9124 - STT_MIB - MARGIN_MIB);
|
||||
assert_eq!(tight.game_room_mib, TOTAL - 1955 - 9124 - STT_MIB - TTS_MIB - MARGIN_MIB);
|
||||
// Other programs already hold the card: no room for a game, the LLM pauses in one.
|
||||
let full = game(TOTAL, OTHERS + 4000, &probe16, 8192);
|
||||
assert!(full.paused);
|
||||
|
||||
@@ -253,8 +253,8 @@ pub trait LlmBrain: Send + Sync {
|
||||
/// `ready` (with whether a game runs) or `paused`.
|
||||
fn llm_state(&self, state: &str, game: bool) -> bool;
|
||||
async fn wait_llm_idle(&self, id: u64, t: std::time::Duration) -> bool;
|
||||
/// The STT worker's model is loaded (on the graphics card).
|
||||
fn stt_on_card(&self) -> bool;
|
||||
/// What the worker's loaded models (STT, TTS) hold on the graphics card, in MiB.
|
||||
fn worker_models_mib(&self) -> i64;
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
@@ -268,8 +268,8 @@ impl LlmBrain for WorkerClient {
|
||||
async fn wait_llm_idle(&self, id: u64, t: std::time::Duration) -> bool {
|
||||
WorkerClient::wait_llm_idle(self, id, t).await
|
||||
}
|
||||
fn stt_on_card(&self) -> bool {
|
||||
self.hello().is_some_and(|h| h.stt_ready)
|
||||
fn worker_models_mib(&self) -> i64 {
|
||||
self.hello().map_or(0, |h| crate::placement::worker_models_mib(h.stt_ready, h.tts_ready))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -320,7 +320,7 @@ async fn perf_loop(settings: crate::settings::Settings, llm: Arc<Llm>, brain: Ar
|
||||
if !brain.wait_llm_idle(id, std::time::Duration::from_secs(120)).await {
|
||||
crate::warn_log!("perf", "Warning: the brain did not finish its answers in 2 minutes; switching anyway");
|
||||
}
|
||||
let ended = match llm.switch(&settings, want, brain.stt_on_card(), game()).await {
|
||||
let ended = match llm.switch(&settings, want, brain.worker_models_mib(), game()).await {
|
||||
Ok(ended) => {
|
||||
retry.succeeded();
|
||||
actual = want;
|
||||
@@ -328,7 +328,7 @@ async fn perf_loop(settings: crate::settings::Settings, llm: Arc<Llm>, brain: Ar
|
||||
}
|
||||
Err(e) => {
|
||||
crate::error_log!("perf", "the LLM did not start in the {want:?} stage: {e}; going back to {actual:?}");
|
||||
let back = match llm.switch(&settings, actual, brain.stt_on_card(), game()).await {
|
||||
let back = match llm.switch(&settings, actual, brain.worker_models_mib(), game()).await {
|
||||
Ok(ended) => ended,
|
||||
Err(e) => {
|
||||
crate::error_log!("perf", "the LLM did not come back in the {actual:?} stage either: {e}");
|
||||
@@ -396,8 +396,8 @@ mod perf_tests {
|
||||
self.0.lock().unwrap().push("waited".into());
|
||||
true
|
||||
}
|
||||
fn stt_on_card(&self) -> bool {
|
||||
false
|
||||
fn worker_models_mib(&self) -> i64 {
|
||||
0
|
||||
}
|
||||
}
|
||||
|
||||
@@ -497,7 +497,7 @@ mod perf_tests {
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn going_back_to_normal_while_the_game_runs_keeps_the_reading_from_before_the_game() {
|
||||
// 24 GB card; other programs 1460 MiB at the start. A game (2500 MiB) starts; the first
|
||||
// 24 GB card; other programs `OTHERS` at the start. A game (2500 MiB) starts; the first
|
||||
// game start fails and the normal stage comes back while the game still runs; the second
|
||||
// try works. It must count other programs from the reading before the game, not one that
|
||||
// has the game in it.
|
||||
@@ -508,7 +508,7 @@ mod perf_tests {
|
||||
let exe = crate::strata::tests::fake_strata(&l, &format!(
|
||||
"case \"$*\" in *'--context 16384'*'expert-vram=0'*) ;; *'--context 16384'*) if [ ! -e '{fo}' ]; then touch '{fo}'; exit 4; fi;; esac; rm -f '{g}'; echo 'strata-q36: (1.10 GiB staging) on 16 CPU threads' >> '{log}'; echo 'Strata is serving.'",
|
||||
fo = failed_once.display(), g = l.data.join("gone").display(), log = log.display()));
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460));
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(crate::strata::tests::OTHERS));
|
||||
let calls = l.data.join("calls");
|
||||
let o2 = others.clone();
|
||||
let llm = Arc::new(Llm::with_exe(l.clone(), exe).with_gpu(Arc::new(move || {
|
||||
@@ -518,7 +518,7 @@ mod perf_tests {
|
||||
})));
|
||||
let s = crate::settings::parse_settings(&serde_json::json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
others.store(1460 + 2500, Ordering::SeqCst);
|
||||
others.store(crate::strata::tests::OTHERS + 2500, Ordering::SeqCst);
|
||||
let brain = Arc::new(FakeBrain::default());
|
||||
let task = tokio::spawn(perf_loop(s, llm.clone(), brain.clone(), Arc::new(|| true), Duration::from_millis(300)));
|
||||
for _ in 0..100 {
|
||||
@@ -529,8 +529,8 @@ mod perf_tests {
|
||||
assert!(failed_once.exists(), "the first game start failed");
|
||||
assert_eq!(llm.stage_view()["stage"], serde_json::json!("gaming"));
|
||||
let probe16 = crate::placement::Probe { context: 16384, parallel: 1, gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 };
|
||||
let right = crate::placement::game(24576, 1460, &probe16, 8192).expert_gib;
|
||||
let wrong = crate::placement::game(24576, 1460 + 2500, &probe16, 8192).expert_gib;
|
||||
let right = crate::placement::game(24576, crate::strata::tests::OTHERS, &probe16, 8192).expert_gib;
|
||||
let wrong = crate::placement::game(24576, crate::strata::tests::OTHERS + 2500, &probe16, 8192).expert_gib;
|
||||
assert_ne!(right, wrong);
|
||||
assert!(launches(&l).last().unwrap().contains(&format!("--set expert-vram={right} ")), "{:?}", launches(&l));
|
||||
}
|
||||
@@ -551,8 +551,8 @@ mod perf_tests {
|
||||
async fn wait_llm_idle(&self, _id: u64, _t: Duration) -> bool {
|
||||
true
|
||||
}
|
||||
fn stt_on_card(&self) -> bool {
|
||||
false
|
||||
fn worker_models_mib(&self) -> i64 {
|
||||
0
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
//! shows how far it got. With `llm.url` the app uses that server instead and starts nothing.
|
||||
|
||||
use std::path::Path;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::atomic::{AtomicBool, AtomicI64, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
@@ -274,9 +274,9 @@ pub struct Llm {
|
||||
/// A game runs (or may run) as the LLM is placed: set before `start` and at every `switch`.
|
||||
/// Only a normal placement made while none runs keeps its reading.
|
||||
game_running: AtomicBool,
|
||||
/// The STT worker's model is on the card (true at switches while the worker runs), so it is not
|
||||
/// taken for another program's memory.
|
||||
stt_on_card: AtomicBool,
|
||||
/// What the voice worker's models (STT, TTS) hold on the card (set at switches while the worker
|
||||
/// runs), so it is not taken for another program's memory.
|
||||
worker_models_mib: AtomicI64,
|
||||
/// The graphics memory a game gets in the placement being started.
|
||||
room: Mutex<Option<i64>>,
|
||||
/// The engine's CPU threads in the normal stage; a game gets half.
|
||||
@@ -302,7 +302,7 @@ impl Llm {
|
||||
pub fn with_exe(layout: Layout, exe: std::path::PathBuf) -> Self {
|
||||
Self { layout, exe, status: Mutex::new(LlmStatus { state: LlmState::Off, managed: false, percent: None, note: None }),
|
||||
active: Mutex::new(None), cancel: Arc::new(AtomicBool::new(false)), port: Mutex::new(0),
|
||||
stop_timeout: Duration::from_secs(120), gpu: None, baseline: Mutex::new(None), game_running: AtomicBool::new(false), stt_on_card: AtomicBool::new(false), room: Mutex::new(None), threads: Mutex::new(None),
|
||||
stop_timeout: Duration::from_secs(120), gpu: None, baseline: Mutex::new(None), game_running: AtomicBool::new(false), worker_models_mib: AtomicI64::new(0), room: Mutex::new(None), threads: Mutex::new(None),
|
||||
stage: Mutex::new((Stage::Normal, None)) }
|
||||
}
|
||||
|
||||
@@ -362,11 +362,10 @@ impl Llm {
|
||||
return Ok(Some(sets));
|
||||
}
|
||||
let Some(card) = self.read_gpu().await else { return Ok(Some(vec![])) };
|
||||
// Other programs: what the card holds without this LLM and without the STT worker's model.
|
||||
// Other programs: what the card holds without this LLM and without the voice worker's models.
|
||||
// The normal stage reads it afresh (programs opened since count); the game stage takes it
|
||||
// from the latest normal placement, as the card then also holds the game.
|
||||
let stt_model = if self.stt_on_card.load(Ordering::SeqCst) { crate::placement::STT_MODEL_MIB } else { 0 };
|
||||
let now_others = (card.used_mib - stt_model).max(0);
|
||||
let now_others = (card.used_mib - self.worker_models_mib.load(Ordering::SeqCst)).max(0);
|
||||
// Only a normal placement made while no game runs keeps its reading (a game's memory would be
|
||||
// taken for other programs'; also on the way back from a failed game stage).
|
||||
if stage != Stage::Gaming && !self.game_running.load(Ordering::SeqCst) {
|
||||
@@ -465,19 +464,20 @@ impl Llm {
|
||||
// A new run: other programs' memory is read afresh.
|
||||
*self.baseline.lock().unwrap() = None;
|
||||
*self.stage.lock().unwrap() = (Stage::Normal, None);
|
||||
self.stt_on_card.store(false, Ordering::SeqCst);
|
||||
self.worker_models_mib.store(0, Ordering::SeqCst);
|
||||
self.start_stage(settings, Stage::Normal).await.map(|r| r.expect("the normal stage never pauses"))
|
||||
}
|
||||
|
||||
/// Move the running LLM to `stage` (`perf.rs`): a restart with that stage's placement, or a stop
|
||||
/// when it pauses (also a game stage with no room for the game). Returns the stage it ended in.
|
||||
/// With `llm.url` nothing moves.
|
||||
/// `stt_on_card`: the STT worker's model is loaded on the card; `game_running`: a game runs now.
|
||||
pub async fn switch(&self, settings: &Settings, stage: Stage, stt_on_card: bool, game_running: bool) -> Result<Stage, String> {
|
||||
/// `worker_models_mib`: what the voice worker's loaded models hold on the card
|
||||
/// (`placement::worker_models_mib`); `game_running`: a game runs now.
|
||||
pub async fn switch(&self, settings: &Settings, stage: Stage, worker_models_mib: i64, game_running: bool) -> Result<Stage, String> {
|
||||
if settings.llm.url.is_some() {
|
||||
return Ok(Stage::Normal);
|
||||
}
|
||||
self.stt_on_card.store(stt_on_card, Ordering::SeqCst);
|
||||
self.worker_models_mib.store(worker_models_mib, Ordering::SeqCst);
|
||||
self.game_running.store(game_running, Ordering::SeqCst);
|
||||
if stage != Stage::Paused && self.start_stage(settings, stage).await?.is_some() {
|
||||
return Ok(stage);
|
||||
@@ -506,6 +506,11 @@ impl Llm {
|
||||
Ok(None) => {}
|
||||
Ok(Some((base, model))) => {
|
||||
crate::info_log!("llm", "ready: {base} model={model}");
|
||||
// What the card has left for everything else once the model is in (the voice
|
||||
// worker's models, a game): a slow answer later can be matched to it.
|
||||
if let Some(g) = self.read_gpu().await {
|
||||
crate::info_log!("llm", "card after the start: {} MiB free of {} MiB", g.total_mib - g.used_mib, g.total_mib);
|
||||
}
|
||||
self.set(LlmState::Ready, None);
|
||||
}
|
||||
Err(e) if e == "cancelled" => self.set(LlmState::Off, None),
|
||||
@@ -649,6 +654,10 @@ pub(crate) mod tests {
|
||||
use crate::datadir::temp_dir;
|
||||
use crate::settings::{parse_settings, ChatChannelSettings};
|
||||
|
||||
/// Other programs on the fake cards: the game PC's 1460 MiB in M0 less the TTS's share, which
|
||||
/// the app keeps beside and M0 did not run, so M0's placements come out.
|
||||
pub(crate) const OTHERS: i64 = 1460 - crate::placement::TTS_MIB;
|
||||
|
||||
pub(crate) fn layout(tag: &str) -> Layout {
|
||||
Layout { data: temp_dir(tag), src: std::path::PathBuf::from("src") }
|
||||
}
|
||||
@@ -881,7 +890,7 @@ pub(crate) mod tests {
|
||||
let log = l.strata_home().join("x.engine.log");
|
||||
let exe = fake_strata(&l, &format!("rm -f '{}'; echo 'strata-q36: hybrid experts: 0 of 256 per layer in VRAM, bigger prompt chunks streamed (1.10 GiB staging); 7.1 s' >> '{}'; echo 'Strata is serving.'",
|
||||
l.data.join("gone").display(), log.display()));
|
||||
// The game PC in M0 (16303 MiB card, 1460 MiB held by other programs); the LLM without
|
||||
// The game PC in M0 (16303 MiB card, `OTHERS` held by other programs); the LLM without
|
||||
// experts held 9874 MiB.
|
||||
let llm = Llm::with_exe(l.clone(), exe).with_gpu(card(&l, "GPU-test", 0));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
@@ -904,7 +913,23 @@ pub(crate) mod tests {
|
||||
assert!(launches[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[0]);
|
||||
}
|
||||
|
||||
/// A fake card for `fake_placed`: 16303 MiB, 1460 MiB held by other programs (and by any
|
||||
#[tokio::test]
|
||||
async fn the_llm_log_tells_how_much_of_the_card_is_free_once_the_model_answers() {
|
||||
crate::logs::install(crate::logs::LogHub::new(None));
|
||||
let l = layout("llm-free");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(card(&l, "GPU-test", 0));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
|
||||
let free = 16303 - OTHERS - 9874;
|
||||
let lines = crate::logs::installed().unwrap()
|
||||
.query(Some("llm"), crate::logs::Level::Info, Some("free"), None, 1000);
|
||||
assert!(lines.iter().any(|x| x.text.contains(&format!("{free} MiB free of 16303 MiB"))),
|
||||
"{:?}", lines.iter().map(|x| &x.text).collect::<Vec<_>>());
|
||||
llm.stop().await.unwrap();
|
||||
}
|
||||
|
||||
/// A fake card for `fake_placed`: 16303 MiB, `OTHERS` held by other programs (and by any
|
||||
/// server still left over, `leftover` MiB, until a `launch --stop`), and 9874 MiB more while a
|
||||
/// Strata server runs (the calls file shows the last launch and stop).
|
||||
#[cfg(unix)]
|
||||
@@ -914,7 +939,7 @@ pub(crate) mod tests {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let stopped = c.contains("launch --stop");
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
let used = 1460 + if running { 9874 } else if stopped { 0 } else { leftover };
|
||||
let used = OTHERS + if running { 9874 } else if stopped { 0 } else { leftover };
|
||||
Some(Gpu { id: id.into(), used_mib: used, total_mib: 16303 })
|
||||
})
|
||||
}
|
||||
@@ -949,7 +974,7 @@ pub(crate) mod tests {
|
||||
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let r2 = reads.clone();
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: 1460, total_mib: 16303 })
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: OTHERS, total_mib: 16303 })
|
||||
}));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
|
||||
@@ -1023,38 +1048,38 @@ pub(crate) mod tests {
|
||||
llm.start(&s).await.unwrap();
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "normal", "gameRoomMib": null}));
|
||||
std::fs::remove_file(l.data.join("calls")).unwrap();
|
||||
assert_eq!(tokio::time::timeout(Duration::from_secs(30), llm.switch(&s, Stage::Gaming, false, true)).await.unwrap().unwrap(), Stage::Gaming);
|
||||
assert_eq!(tokio::time::timeout(Duration::from_secs(30), llm.switch(&s, Stage::Gaming, 0, true)).await.unwrap().unwrap(), Stage::Gaming);
|
||||
let g = launches(&l);
|
||||
// Measured once at the game's context and one request, then the 2 GiB minimum (8 GB does not
|
||||
// fit on 16 GB) with half the engine's 16 CPU threads.
|
||||
assert_eq!(g.len(), 2, "{g:?}");
|
||||
assert!(g[0].contains("--context 16384") && g[0].contains("--parallel 1") && g[0].contains("--set expert-vram=0"), "{}", g[0]);
|
||||
assert!(g[1].ends_with("--set expert-vram=2 --set prefill=2048 --set pool-workers=8"), "{}", g[1]);
|
||||
let room = 16303 - 1460 - 9874 - crate::placement::STT_MIB - crate::placement::MARGIN_MIB - 921;
|
||||
let room = 16303 - OTHERS - 9874 - crate::placement::STT_MIB - crate::placement::TTS_MIB - crate::placement::MARGIN_MIB - 921;
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "gaming", "gameRoomMib": room}));
|
||||
assert_eq!(llm.status().state, LlmState::Ready);
|
||||
// Back to normal: the run's first reading of other programs, the normal placement again.
|
||||
std::fs::remove_file(l.data.join("calls")).unwrap();
|
||||
assert_eq!(llm.switch(&s, Stage::Normal, false, false).await.unwrap(), Stage::Normal);
|
||||
assert_eq!(llm.switch(&s, Stage::Normal, 0, false).await.unwrap(), Stage::Normal);
|
||||
let n = launches(&l);
|
||||
assert_eq!(n.len(), 1, "{n:?}");
|
||||
assert!(n[0].contains("--context 32768") && n[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", n[0]);
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "normal", "gameRoomMib": null}));
|
||||
// Pause: the LLM stops.
|
||||
assert_eq!(llm.switch(&s, Stage::Paused, false, true).await.unwrap(), Stage::Paused);
|
||||
assert_eq!(llm.switch(&s, Stage::Paused, 0, true).await.unwrap(), Stage::Paused);
|
||||
assert_eq!(llm.status().state, LlmState::Off);
|
||||
assert_eq!(llm.stage_view()["stage"], json!("paused"));
|
||||
}
|
||||
|
||||
/// A card whose other programs hold `others` MiB (changeable), plus the STT model while
|
||||
/// `stt` is set, plus 9874 MiB while a Strata server runs.
|
||||
/// A card whose other programs hold `others` MiB (changeable), plus the voice worker's models
|
||||
/// (`worker` MiB), plus 9874 MiB while a Strata server runs.
|
||||
#[cfg(unix)]
|
||||
fn live_card(l: &Layout, others: Arc<std::sync::atomic::AtomicI64>, stt: bool) -> GpuReader {
|
||||
fn live_card(l: &Layout, others: Arc<std::sync::atomic::AtomicI64>, worker: i64) -> GpuReader {
|
||||
let calls = l.data.join("calls");
|
||||
Arc::new(move || {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
let used = others.load(Ordering::SeqCst) + if stt { 1300 } else { 0 } + if running { 9874 } else { 0 };
|
||||
let used = others.load(Ordering::SeqCst) + worker + if running { 9874 } else { 0 };
|
||||
Some(Gpu { id: "GPU-test".into(), used_mib: used, total_mib: 16303 })
|
||||
})
|
||||
}
|
||||
@@ -1064,25 +1089,48 @@ pub(crate) mod tests {
|
||||
async fn the_normal_placement_reads_other_programs_afresh_and_a_game_is_not_counted_twice() {
|
||||
let l = layout("llm-stage-fresh");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460));
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(OTHERS));
|
||||
// At the start the STT worker has not loaded; at every switch it has.
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card(&l, others.clone(), false));
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card(&l, others.clone(), 0));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
let llm = llm.with_gpu(live_card(&l, others.clone(), true));
|
||||
let stt = crate::placement::worker_models_mib(true, false);
|
||||
let llm = llm.with_gpu(live_card(&l, others.clone(), stt));
|
||||
// A game starts and holds 2500 MiB: the game stage counts other programs as before the game.
|
||||
others.store(1460 + 2500, Ordering::SeqCst);
|
||||
llm.switch(&s, Stage::Gaming, true, true).await.unwrap();
|
||||
others.store(OTHERS + 2500, Ordering::SeqCst);
|
||||
llm.switch(&s, Stage::Gaming, stt, true).await.unwrap();
|
||||
assert!(launches(&l).last().unwrap().contains("--set expert-vram=2 "), "{:?}", launches(&l));
|
||||
// The game ends but a video player now holds 1000 MiB more: the normal stage reads that.
|
||||
others.store(1460 + 1000, Ordering::SeqCst);
|
||||
llm.switch(&s, Stage::Normal, true, false).await.unwrap();
|
||||
let expected = crate::placement::normal(16303, 1460 + 1000, &crate::placement::Probe { context: 32768, parallel: 1,
|
||||
others.store(OTHERS + 1000, Ordering::SeqCst);
|
||||
llm.switch(&s, Stage::Normal, stt, false).await.unwrap();
|
||||
let expected = crate::placement::normal(16303, OTHERS + 1000, &crate::placement::Probe { context: 32768, parallel: 1,
|
||||
gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 }, 1024);
|
||||
assert!(expected < 1.5);
|
||||
assert!(launches(&l).last().unwrap().ends_with(&format!("--set expert-vram={expected} --set prefill=2048")), "{:?}", launches(&l));
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn the_voice_models_loaded_after_the_start_change_nothing_on_the_way_back() {
|
||||
// At the start neither the STT nor the TTS has loaded; at the switches both hold the card.
|
||||
// With nothing else changed, the normal stage comes back with the placement it started with.
|
||||
let l = layout("llm-stage-voice");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(460));
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card(&l, others.clone(), 0));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
let first = launches(&l).last().unwrap().clone();
|
||||
let voice = crate::placement::worker_models_mib(true, true);
|
||||
let held = crate::placement::STT_MODEL_MIB + crate::placement::TTS_MIB;
|
||||
let llm = llm.with_gpu(live_card(&l, others.clone(), held));
|
||||
llm.switch(&s, Stage::Gaming, voice, true).await.unwrap();
|
||||
llm.switch(&s, Stage::Normal, voice, false).await.unwrap();
|
||||
let back = launches(&l).last().unwrap().clone();
|
||||
assert!(!first.contains("--set expert-vram=0 "), "the test has room for experts: {first}");
|
||||
assert_eq!(back, first);
|
||||
}
|
||||
|
||||
/// `live_card` on a card of `total` MiB.
|
||||
#[cfg(unix)]
|
||||
fn live_card_of(l: &Layout, others: Arc<std::sync::atomic::AtomicI64>, total: i64) -> GpuReader {
|
||||
@@ -1100,33 +1148,33 @@ pub(crate) mod tests {
|
||||
// A game holds 2500 MiB already: it is in what the card holds, so no reserve on top.
|
||||
let l = layout("llm-stage-game-at-start");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460 + 2500));
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(OTHERS + 2500));
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card_of(&l, others.clone(), 16303));
|
||||
llm.game_at_start(true);
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming, false, true).await.unwrap(), Stage::Gaming, "not paused");
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Gaming, "not paused");
|
||||
let probe16 = crate::placement::Probe { context: 16384, parallel: 1, gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 };
|
||||
let expected = crate::placement::normal(16303, 1460 + 2500, &probe16, 1024);
|
||||
let expected = crate::placement::normal(16303, OTHERS + 2500, &probe16, 1024);
|
||||
assert!(launches(&l).last().unwrap().contains(&format!("--set expert-vram={expected} ")), "{:?}", launches(&l));
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn the_game_stage_reads_the_card_again_and_counts_what_grew_since_the_normal_reading() {
|
||||
// Normal reading: 1460 MiB. Then a program takes 4000 MiB and the game 4500 MiB: together
|
||||
// Normal reading: OTHERS. Then a program takes 4000 MiB and the game 4500 MiB: together
|
||||
// more than the 8 GB reserve, so the placement leaves that much, not just 8 GB.
|
||||
let l = layout("llm-stage-grew");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(1460));
|
||||
let others = Arc::new(std::sync::atomic::AtomicI64::new(OTHERS));
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(live_card_of(&l, others.clone(), 24576));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
others.store(1460 + 4000 + 4500, Ordering::SeqCst);
|
||||
llm.switch(&s, Stage::Gaming, false, true).await.unwrap();
|
||||
others.store(OTHERS + 4000 + 4500, Ordering::SeqCst);
|
||||
llm.switch(&s, Stage::Gaming, 0, true).await.unwrap();
|
||||
let probe16 = crate::placement::Probe { context: 16384, parallel: 1, gpu: "GPU-test".into(), model_mib: 9874, staging_gib: 1.1 };
|
||||
let expected = crate::placement::game(24576, 1460, &probe16, 8500).expert_gib;
|
||||
assert_ne!(expected, crate::placement::game(24576, 1460, &probe16, 8192).expert_gib, "the test tells them apart");
|
||||
let expected = crate::placement::game(24576, OTHERS, &probe16, 8500).expert_gib;
|
||||
assert_ne!(expected, crate::placement::game(24576, OTHERS, &probe16, 8192).expert_gib, "the test tells them apart");
|
||||
assert!(launches(&l).last().unwrap().contains(&format!("--set expert-vram={expected} ")), "{:?}", launches(&l));
|
||||
}
|
||||
|
||||
@@ -1143,9 +1191,9 @@ pub(crate) mod tests {
|
||||
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let r2 = reads.clone();
|
||||
let llm = llm.with_gpu(Arc::new(move || {
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: 1460, total_mib: 16303 })
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: OTHERS, total_mib: 16303 })
|
||||
}));
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming, false, true).await.unwrap(), Stage::Gaming);
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Gaming);
|
||||
assert!(launches(&l).last().unwrap().contains("--set expert-vram=0 "), "{:?}", launches(&l));
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "gaming", "gameRoomMib": null}));
|
||||
}
|
||||
@@ -1160,11 +1208,11 @@ pub(crate) mod tests {
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
Some(Gpu { id: "GPU-small".into(), used_mib: 1460 + if running { 9874 } else { 0 }, total_mib: 12288 })
|
||||
Some(Gpu { id: "GPU-small".into(), used_mib: OTHERS + if running { 9874 } else { 0 }, total_mib: 12288 })
|
||||
}));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming, false, true).await.unwrap(), Stage::Paused);
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Paused);
|
||||
assert_eq!(llm.status().state, LlmState::Off);
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "paused", "gameRoomMib": null}));
|
||||
let c = std::fs::read_to_string(l.data.join("calls")).unwrap();
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
|---|---|
|
||||
| 독점 전체화면 | Windows가 알려 주는 "전체화면 D3D 앱 실행 중" 상태로 감지합니다. |
|
||||
| 테두리 없는 창 모드 (borderless) | 앞에 있는 창이 아래 세 조건을 모두 만족하면 게임으로 봅니다. 셋 다 맞아야 하므로 일반 프로그램을 최대화한 것과 구분됩니다. (1) 모니터 한 장을 빈틈없이 덮습니다. (2) 제목 표시줄·테두리가 없습니다. (3) 그 프로세스가 GPU 3D 엔진을 일정 수준 이상 씁니다. |
|
||||
| 게임 폴더·목록 | Steam, Epic, Riot, Battle.net 게임 폴더에서 실행된 프로그램과 GUI에서 직접 등록한 게임은 창 모양과 상관없이 게임으로 봅니다. Battle.net 게임은 아무 폴더에나 설치되므로 Windows에 등록된 설치 정보(프로그램 제거 목록)에서 설치 폴더를 찾습니다. |
|
||||
| 게임 폴더·목록 | Steam, Epic, Riot, Battle.net 게임 폴더에서 실행된 프로그램과 GUI에서 직접 등록한 게임은 창 모양과 상관없이 게임으로 봅니다. 단, 런처 자신의 폴더와 게임이 아닌 앱의 폴더(Steam의 `Steamworks Shared`·`wallpaper_engine`, Epic의 `Launcher`·`Epic Online Services`, Riot의 `Riot Client`)는 빼서, 늘 켜져 있는 Wallpaper Engine이나 Riot 클라이언트 크래시 핸들러를 게임으로 잡지 않습니다(2026-11-17). Battle.net 게임은 아무 폴더에나 설치되므로 Windows에 등록된 설치 정보(프로그램 제거 목록)에서 설치 폴더를 찾습니다. |
|
||||
| 게임이 아닌 것 | 브라우저 전체화면 영상, 동영상 플레이어, 바탕화면, 디스코드, 봇용 크롬은 제외합니다. 잘못 잡히는 프로그램은 "게임 아님" 목록에 넣습니다. |
|
||||
|
||||
한 번 게임으로 감지되면 게임 프로세스가 끝날 때까지 게임 모드를 유지합니다. Alt+Tab으로 다른 창을 봐도 마찬가지입니다. 게임이 뒤에 있어도 VRAM은 계속 쓰기 때문입니다. 대시보드에는 "게임 모드: 켜짐 (게임 이름, 감지 이유)"를 표시합니다.
|
||||
@@ -31,7 +31,7 @@
|
||||
|
||||
### PC마다 값을 계산
|
||||
|
||||
- **배치는 계산해서 정함**: 전문가 층 VRAM 몫 = 그래픽카드 VRAM − 게임용으로 비울 VRAM − 다른 프로그램이 이미 쓰는 VRAM − STT 몫(약 2.7GB: 모델 1.3GB와 10명 동시 전사 때 늘어나는 1.44GB, M0 `stt` 실측) − 엔진 기본 버퍼 − 여유분(768MiB, 엔진이 스스로 남기는 양과 같음). 평소 배치도 같은 식이고, 게임용으로 비울 VRAM 대신 다른 프로그램용 여유(기본 1GB)를 뺍니다(사용자 결정, 2026-11-10). 이 여유는 나중에 켜는 브라우저 영상 같은 프로그램 몫이고, 이미 켜져 있는 프로그램은 "다른 프로그램이 이미 쓰는 VRAM"으로 따로 빠집니다. 게임 중에는 게임용으로 비울 VRAM이 이 역할을 하므로 다른 프로그램용 여유를 따로 빼지 않습니다. 엔진 기본 버퍼는 첫 실행 자가 점검에서 전문가 층 없이 한 번 켜서 잽니다. 정해 둔 숫자가 아니라 이 식으로 구하므로 VRAM이 작은 PC에도 같은 규칙이 맞습니다.
|
||||
- **배치는 계산해서 정함**: 전문가 층 VRAM 몫 = 그래픽카드 VRAM − 게임용으로 비울 VRAM − 다른 프로그램이 이미 쓰는 VRAM − STT 몫(약 2.7GB: 모델 1.3GB와 10명 동시 전사 때 늘어나는 1.44GB, M0 `stt` 실측) − TTS 몫(1005MiB, MeloTTS 워밍업 합성 뒤, M0 `tts_melo` 실측; LLM보다 늦게 올라오므로 미리 남겨야 처음 켤 때와 전환할 때 값이 같음, 2026-11-17) − 엔진 기본 버퍼 − 여유분(768MiB, 엔진이 스스로 남기는 양과 같음). 평소 배치도 같은 식이고, 게임용으로 비울 VRAM 대신 다른 프로그램용 여유(기본 1GB)를 뺍니다(사용자 결정, 2026-11-10). 이 여유는 나중에 켜는 브라우저 영상 같은 프로그램 몫이고, 이미 켜져 있는 프로그램은 "다른 프로그램이 이미 쓰는 VRAM"으로 따로 빠집니다. 게임 중에는 게임용으로 비울 VRAM이 이 역할을 하므로 다른 프로그램용 여유를 따로 빼지 않습니다. 엔진 기본 버퍼는 첫 실행 자가 점검에서 전문가 층 없이 한 번 켜서 잽니다. 정해 둔 숫자가 아니라 이 식으로 구하므로 VRAM이 작은 PC에도 같은 규칙이 맞습니다.
|
||||
- **게임 중 최소 배치는 전문가 2GB입니다**(사용자 결정, 2026-11-10). 전문가 몫에는 긴 프롬프트용 버퍼 1.1GiB가 먼저 들어가므로 2GB는 전문가를 실제로 올리는 가장 작은 값이고, 실제 게임 중에는 0GB보다 출력이 1.5~2.1배 빨랐습니다(아래 측정 결과). 0GB는 2GB로는 게임 몫이 0 이하가 되는 PC에서만 일시정지 전 마지막 단계로 씁니다.
|
||||
- **몫이 최소 배치보다 작을 때**: 계산한 몫이 최소 배치(전문가 2GB)보다 작으면, 몫을 늘려 게임 몫을 침범하지 않습니다. 대신 게임용으로 비울 VRAM을 "그래픽카드 VRAM − 다른 프로그램 − STT 몫 − 엔진 기본 버퍼 − 최소 배치"까지 줄이고, 대시보드와 게임 모드 설정 화면에 "이 PC에서는 게임에 X GB까지만 비울 수 있음"을 표시합니다. 이렇게 줄인 값이 0 이하이면(예: 8GB 그래픽카드에서 다른 프로그램이 이미 많이 씀) 게임 중에는 LLM을 멈추고 일시정지 단계로 갑니다. 게임 중 일시정지를 꺼 두었어도 마찬가지이고, 그 이유를 대시보드에 표시합니다. 게임이 끝나면 되돌리기 규칙대로 돌아옵니다.
|
||||
- **첫 실행 자가 점검**: 설치가 끝나면 평소·게임 중 배치로 각각 30초씩 출력 속도와 음성 첫 문장 시간을 잽니다. 평소 배치가 기준(출력 30 tok/s, 음성 첫 문장 0.7초)에 못 미치는 PC는 평소에도 이미지 CPU 읽기, 지난 대화 약 4천 토큰으로 시작합니다. 게임 중 배치가 기준에 못 미치면 대시보드와 설정 화면에 "게임 중에는 답이 느릴 수 있음"을 표시합니다.
|
||||
|
||||
File diff suppressed because one or more lines are too long
78
src/jarvis/tools/builtin/pc_control.spec.md
Normal file
78
src/jarvis/tools/builtin/pc_control.spec.md
Normal file
@@ -0,0 +1,78 @@
|
||||
# pcControl tool spec (v2 M7)
|
||||
|
||||
Lets Jarvis operate programs on this PC for a registered person: list the windows, read a window's
|
||||
buttons and fields, click, type, press keys, start and close programs ("메모장 열어서 장보기 목록
|
||||
적어 줘", "계산기로 12 더하기 3 해 줘"). Product rules: `docs/v2/pc-features.md` ("키보드·마우스
|
||||
제어와 안티치트"), `docs/v2/proactive.md` (제안 3, user decision 2026-11-14). The app does every
|
||||
action through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"); this tool
|
||||
asks it, holds the next-message consent and returns raw text.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Only in v2, only when the app offers it.** The tool needs a PC backend in the turn state
|
||||
(`turn_state.set_pc`, every v2 voice and chat reply; not scheduled tasks). The app puts
|
||||
`pcControl` in the brain's `disabled_tools` unless PC control is available (Windows, the feature
|
||||
on, someone in `pcControl.allowedUsers`, agent-desktop installed). Without a backend it refuses.
|
||||
- **The app decides who may and what.** Every action goes to the app with the asker. People not
|
||||
registered, an action turned off (`never`), an anti-cheat game running, the emergency stop, a
|
||||
password field or a window run as administrator: the app refuses and the tool returns its reason
|
||||
as an error, which the model says plainly. The tool never retries a refusal another way.
|
||||
- **Asking first is the person's own next message.** For an action at `ask` (by default click,
|
||||
type, keys, run, close) the first call does nothing: the app answers that consent is needed, the
|
||||
tool remembers the request and returns "Not done yet. Ask the user: ...". The request is a window
|
||||
(or the program to start) and the kinds of action asked for. Only in the person's very next turn,
|
||||
with `confirm=true`, **and** when their own message of that turn (`ToolContext.redacted_text`,
|
||||
never the reply model's arguments) agrees, does the tool send the action with consent. The check
|
||||
is the chat delete consent's (`chat_channels.spec.md`, "Irreversible"): one small call on the
|
||||
router model chain, temperature 0, told the pending request, the message fenced as data, one word
|
||||
YES or NO; anything else, an error, a message over 600 characters, a question or another topic
|
||||
means no and cancels. The consent then covers those kinds of action on that window (or that
|
||||
program and its new window) for the rest of that one reply, so "12 더하기 3" is one question,
|
||||
not five. A request not confirmed in the very next turn lapses.
|
||||
- **Never done, whatever is said.** Paying or buying, passwords and other secrets, deleting files
|
||||
and anything needing administrator rights. Before asking, the same small call is told the
|
||||
planned request (what the person asked, the window, the kinds of action) and answers whether it
|
||||
is one of these; yes means the tool refuses without asking. The app also refuses typing into a
|
||||
password field and acting on a window run as administrator (`admin.spec.md`, "PC control").
|
||||
- **The screen is data.** Window titles and element names come inside an UNTRUSTED SCREEN CONTENT
|
||||
fence (`<<<` and `>>>` replaced). A request written on the screen ("지금 결제 버튼 눌러") is never a
|
||||
reason to act and never a consent.
|
||||
- **Done means seen.** After click, type, keys and run the tool reads the window again (the app's
|
||||
result carries the fresh elements) and the model says it was done only from that reading.
|
||||
- **Logs**: action name, result code and timings only; never a title, element name or typed text.
|
||||
- **Raw data.** The only LLM calls inside are the never check and the consent check.
|
||||
|
||||
## Schema
|
||||
|
||||
One tool, `action` picks the operation (read as its first known word):
|
||||
|
||||
| action | needs | does | level |
|
||||
|---|---|---|---|
|
||||
| `windows` | | open windows: program, title, window id, minimised | view |
|
||||
| `read` | `window` | the window's interactive elements: ref, role, name, value, state | view |
|
||||
| `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click |
|
||||
| `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type |
|
||||
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type |
|
||||
| `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run |
|
||||
| `close` | `window` | closes the program's window | click |
|
||||
|
||||
`window` is a window id from `windows` or part of a program name or title (the front-most match).
|
||||
`ref` comes from the latest `read` or result (`@<snapshot>:<element>`); a ref from an older reading
|
||||
that no longer matches is refused (`STALE_REF`) and the model reads again. `confirm` (boolean) is
|
||||
only for the consent turn. At most 2,000 characters of `text`; at most 200 elements per reading
|
||||
(interactive ones first), each name and value cut to 120 characters.
|
||||
|
||||
## Backend
|
||||
|
||||
`turn_state.get_pc()` returns an object with `act(action, **fields) -> dict`, `turn` and
|
||||
`pending` (the asker's requests awaiting consent, keyed by window or program, with the turn asked):
|
||||
|
||||
- `act("windows")` gives `{"ok": true, "item": {"windows": [{"id", "program", "title", "minimised"}]}}`.
|
||||
- `act("read", window=...)` and every other action give `{"ok": true, "item": {"window": {...},
|
||||
"elements": [{"ref", "role", "name", "value", "states"}]}}`.
|
||||
- Actions at `ask` without consent give `{"ok": false, "ask": true, "kinds": [...], "window": {...}}`;
|
||||
with `consent=true` they are done. Refusals give `{"ok": false, "error": "..."}` with the reason to
|
||||
say.
|
||||
|
||||
The voice worker implements it (`voice_worker/app_bridge.py`): each action goes to the app as
|
||||
`pc_action` with the asker (`voice_worker.spec.md`, "PC control"). Eval: `evals/test_pc_control_tool.py`.
|
||||
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
from .log import debug_log
|
||||
@@ -42,7 +43,7 @@ def main() -> int:
|
||||
from .brain import take_mcp_env
|
||||
mcp_env = take_mcp_env()
|
||||
|
||||
from .server import VoiceWorker, is_loopback
|
||||
from .server import VoiceWorker, is_loopback, nvidia_card
|
||||
from .stt_whisper import WhisperTranscriber
|
||||
from .vad import SileroVad
|
||||
|
||||
@@ -81,7 +82,8 @@ def main() -> int:
|
||||
recorder = Recorder(os.environ[RECORDINGS_ENV])
|
||||
|
||||
async def run() -> None:
|
||||
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain, recorder=recorder)
|
||||
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain, recorder=recorder,
|
||||
card=nvidia_card if shutil.which("nvidia-smi") else None)
|
||||
server = await worker.serve(args.host, args.port)
|
||||
debug_log(f"voice worker listening on {args.host}:{args.port} tts={'on' if tts else 'off'} "
|
||||
f"history={'on' if history else 'off'} brain={'on' if brain else 'off'} "
|
||||
|
||||
@@ -15,7 +15,7 @@ from typing import Callable
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .log import debug_log
|
||||
from .log import debug_log, notice
|
||||
|
||||
from .audio import InputConverter, loudness_equalise
|
||||
from .brain import Brain, MemorySettings
|
||||
@@ -44,6 +44,19 @@ def is_loopback(host: str) -> bool:
|
||||
LLM_WAIT_S = 180.0
|
||||
|
||||
|
||||
def nvidia_card() -> tuple[int, int] | None:
|
||||
"""The first NVIDIA card's used and total memory in MiB (``nvidia-smi``), None when unreadable."""
|
||||
import subprocess
|
||||
try:
|
||||
out = subprocess.run(["nvidia-smi", "--query-gpu=memory.used,memory.total", "--format=csv,noheader,nounits"],
|
||||
capture_output=True, text=True, timeout=5,
|
||||
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0)).stdout
|
||||
used, total = (int(v) for v in out.splitlines()[0].split(","))
|
||||
return used, total
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
class _LlmPaused(Exception):
|
||||
"""The app paused the LLM (a game runs): nothing is answered until it is ready again."""
|
||||
|
||||
@@ -53,6 +66,7 @@ MAX_NOTES = 20
|
||||
MAX_NOTE_CHARS = 300
|
||||
# How long answer traces stay (history.spec.md, "Traces"); removed when the worker starts.
|
||||
TRACE_KEEP_DAYS = 30
|
||||
CARD_EVERY_S = 1.0 # how often the card's free memory is read while an answer is written
|
||||
|
||||
|
||||
class _LiveTrace(list):
|
||||
@@ -93,7 +107,8 @@ class VoiceWorker:
|
||||
def __init__(self, token: str, transcriber: Transcriber | None, vad_factory: Callable[[], VadStream],
|
||||
tts_engine: TtsEngine | None = None, hello_timeout_s: float = 5.0, tick_s: float = 0.05,
|
||||
history=None, brain: Brain | None = None, chat_timeout_s: float = 15.0,
|
||||
delivery_timeout_s: float = 30.0, recorder=None):
|
||||
delivery_timeout_s: float = 30.0, recorder=None,
|
||||
card: Callable[[], tuple[int, int] | None] | None = None):
|
||||
if not token:
|
||||
raise ValueError("a connection token is required")
|
||||
self.token = token
|
||||
@@ -105,6 +120,7 @@ class VoiceWorker:
|
||||
self.history = history # jarvis.memory.history.HistoryStore or None
|
||||
self.recorder = recorder # recordings.Recorder (voice.record on) or None
|
||||
self.brain = brain
|
||||
self.card = card # (used MiB, total MiB) of the graphics card, or None without one
|
||||
# One reply at a time per lane: voice answers never wait behind another person's chat answer
|
||||
# (conversation.md: with an "answer everything" chat channel the LLM takes two or more
|
||||
# requests at once). One person's replies stay one at a time across lanes, so their dialogue
|
||||
@@ -687,7 +703,7 @@ class Session:
|
||||
history = HistoryContext()
|
||||
try:
|
||||
trace.append(self._records_step(history, notes))
|
||||
async with self._llm():
|
||||
async with self._llm(), self._card_watch(f"task {rid}"):
|
||||
answer = await asyncio.to_thread(w.brain.reply, user, text, history, task=True,
|
||||
notes=notes or [], trace=trace)
|
||||
except Exception as e: # the brain must never take the session down
|
||||
@@ -752,6 +768,34 @@ class Session:
|
||||
w.llm_active -= 1
|
||||
self._llm_check_idle()
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _card_watch(self, label: str):
|
||||
"""While an answer is written: the card's free memory, read every `CARD_EVERY_S`; the least
|
||||
seen is logged once it ends (a slow answer can then be matched to a full card)."""
|
||||
card = self.worker.card
|
||||
if card is None:
|
||||
yield
|
||||
return
|
||||
seen: list[tuple[int, int]] = []
|
||||
|
||||
async def read():
|
||||
while True:
|
||||
got = await asyncio.to_thread(card)
|
||||
if got:
|
||||
seen.append(got)
|
||||
await asyncio.sleep(CARD_EVERY_S)
|
||||
|
||||
t0 = time.monotonic()
|
||||
task = asyncio.get_running_loop().create_task(read())
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
task.cancel()
|
||||
if seen:
|
||||
total = seen[-1][1]
|
||||
notice(f"{label} card: least {min(total - u for u, _ in seen)} MiB free of {total} MiB "
|
||||
f"({len(seen)} readings, {int((time.monotonic() - t0) * 1000)} ms)", "llm")
|
||||
|
||||
def _on_prepare(self, msg: dict) -> None:
|
||||
"""A call still being said: the brain gets ready for it. One prepare per user runs at a time;
|
||||
while it runs only the newest text waits, and one about the same as the last one prepared
|
||||
@@ -909,7 +953,7 @@ class Session:
|
||||
screen = AppBridge(self, asyncio.get_running_loop(), "screen", *origin)
|
||||
trace.append(self._records_step(history, notes))
|
||||
try:
|
||||
async with self._llm(rid if speak else None):
|
||||
async with self._llm(rid if speak else None), self._card_watch(f"answer {rid}"):
|
||||
w.answering += 1
|
||||
try:
|
||||
answer = await asyncio.to_thread(w.brain.reply, user, text, history, chat=chat,
|
||||
|
||||
@@ -410,6 +410,23 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
|
||||
dropped. The pictures stay in that one reply (screen_view.spec.md).
|
||||
- Messages may be up to 4 MiB (pictures), the WebSocket's limit.
|
||||
|
||||
### PC control (M7)
|
||||
|
||||
The brain's pcControl tool (`src/jarvis/tools/builtin/pc_control.spec.md`) asks the app, which acts
|
||||
through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"):
|
||||
|
||||
```
|
||||
<- {"type": "pc_action", "id": "p1", "action": "windows|read|click|type|keys|run|close", "user": "<id>",
|
||||
"via": "voice|chat", "channel": "...", "window": "...", "ref": "@s1:e8", "text": "...", "keys": "ctrl+s",
|
||||
"program": "notepad", "consent": true}
|
||||
-> {"type": "pc_action_result", "id": "p1", "ok": true, "item": {...}} // or ok false with "error", or "ask": true
|
||||
```
|
||||
|
||||
- Every voice and chat reply gets a PC backend for its asker (not scheduled tasks); ids and waiting
|
||||
are those of chat and schedule actions. The worker keeps each asker's requests awaiting consent
|
||||
with the turn they were asked in, numbered like the chat delete confirmations ("History and
|
||||
brain"), so only a message sent after the question counts as the next turn.
|
||||
|
||||
### Background collection (M6)
|
||||
|
||||
With `JARVIS_COLLECT_PATH` set, the worker collects on its own thread, apart from the voice and the
|
||||
@@ -514,3 +531,10 @@ cancel, and engine failures. The category names the service, so the app's log sc
|
||||
in its tab (`app/src/admin.spec.md`, "Logs"): `stt` (utterance ends, STT batches, the Whisper model),
|
||||
`tts` (sentences, cancels, MeloTTS), `llm` (the brain's replies, prepares and judgements),
|
||||
`voice_worker` for the rest (connections, sessions, channels).
|
||||
|
||||
Without debug logging too, each answer and each scheduled task logs one `llm` line once the LLM is
|
||||
done with it: the least free graphics memory read while it was written, the card's total, how many
|
||||
readings and how long it took (`answer <id> card: least N MiB free of T MiB (k readings, ms)`). The
|
||||
card is read with `nvidia-smi` once a second while the answer is written, in a thread, so the answer
|
||||
never waits for it; without `nvidia-smi` nothing is read or logged. The line carries no text, so a
|
||||
slow answer can be matched to a full card (the engine log has the same answer's timings).
|
||||
|
||||
@@ -1568,3 +1568,51 @@ def test_the_judgement_is_told_who_is_in_the_channel(monkeypatch):
|
||||
assert "Several people" in unknown and "anything unclear" in unknown, "an app that does not say: as before"
|
||||
assert "one other person" in pair and "unclear, answer yes" in pair
|
||||
assert "5 people" in group and "Only 1 of the 5" in group and "anything unclear" in group
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_each_answer_logs_the_least_graphics_memory_free_while_it_was_written(store, monkeypatch, capfd):
|
||||
"""voice_worker.spec.md, "Debug logging": an answer (and a scheduled task) logs the least free
|
||||
graphics memory read while the LLM wrote it, without debug logging and without its text, so a
|
||||
slow answer can be matched to a full card."""
|
||||
import time as _time
|
||||
from voice_worker import server as worker_server
|
||||
monkeypatch.setattr(worker_server, "CARD_EVERY_S", 0.02)
|
||||
readings = iter([9000, 15000, 12000])
|
||||
last = [12000]
|
||||
|
||||
def card():
|
||||
last[0] = next(readings, last[0])
|
||||
return last[0], 16303
|
||||
|
||||
def slow(user, text):
|
||||
_time.sleep(0.3)
|
||||
return f"secret answer for {text}"
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="reply", id="r1", user="100", text="private question", via="chat", speak=False)
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
await send(ws, type="reply", id="t1", user="100", text="private task", via="task")
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text" and m["id"] == "t1")
|
||||
|
||||
asyncio.run(with_worker(body, store=store, brain=FakeBrain(slow), card=card))
|
||||
lines = [x for x in capfd.readouterr().err.splitlines() if x.startswith(f"[{'llm':^10}]") and "free" in x]
|
||||
reply = [x for x in lines if "answer r1" in x]
|
||||
task = [x for x in lines if "task t1" in x]
|
||||
assert len(reply) == 1 and len(task) == 1, lines
|
||||
assert f"least {16303 - 15000} MiB free of 16303 MiB" in reply[0], reply
|
||||
assert "private" not in "".join(lines) and "secret" not in "".join(lines)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_without_a_card_reading_an_answer_logs_no_card_line(store, capfd):
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="reply", id="r1", user="100", text="q", via="chat", speak=False)
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
|
||||
asyncio.run(with_worker(body, store=store, brain=FakeBrain(), card=lambda: None))
|
||||
assert "MiB free" not in capfd.readouterr().err
|
||||
|
||||
Reference in New Issue
Block a user