Compare commits

..

5 Commits

Author SHA1 Message Date
javis-bot
1c06b16bd4 chore(app): version 0.1.25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
44387685af fix(app): log the card's real free memory, and short answers get their line too
Free memory is nvidia-smi's memory.free in the app and the voice worker, so the
driver's reserved share no longer counts as free. A card reading still running
when an answer ends finishes on its own and then logs, so an answer shorter
than one reading is logged too, and the answer still never waits for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
61b68130c6 chore(app): version 0.1.24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:48 +09:00
javis-bot
f1600f09c2 feat(app): log the graphics card's free memory after the LLM starts and while each answer is written
The app logs how much of the card is free once the model answers, and the
voice worker reads the card once a second while an answer or a scheduled task
is written and logs the least free memory seen (no text, llm tab), so a slow
answer can be matched to a full card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:20 +09:00
javis-bot
ce03424ba4 docs(spec): PC control tool, next-message consent, agent-desktop in the app
pcControl (v2 M7): windows, read, click, type, keys, run, close through the
app's agent-desktop. Ask-level actions wait for the asker's own next message
and then cover that window's asked kinds for the reply; payments, passwords,
deleting files and admin rights are filtered before asking and never done.
The app refuses during an anti-cheat game and the emergency stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 14:04:43 +09:00
14 changed files with 342 additions and 22 deletions

View File

@@ -27,6 +27,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
| `src/jarvis/tools/builtin/personal_notes.spec.md` | personalNotes tool (v2): each person's standing instructions for how they are answered; the app keeps them and sends the asker's with every reply | Offered only with a notes backend; only the asker's own, only in their answers; fenced, never permits an action |
| `src/jarvis/tools/builtin/collected_news.spec.md` | collectedNews tool (v2): read the background collection's results (targets, newest items) | Offered only with the app's collection file and the feature on; web content fenced as data; read-only, no LLM |
| `src/jarvis/tools/builtin/screen_view.spec.md` | screenView tool (v2): see the whole screen, one monitor or one window through the app; pictures go to the reply model for that reply only | Offered only when the app makes screen viewing available; the app decides who may see; screen text is data; pictures never kept or logged |
| `src/jarvis/tools/builtin/pc_control.spec.md` | pcControl tool (v2 M7): list windows, read a window, click, type, keys, run and close programs through the app's agent-desktop | Offered only when the app makes PC control available; ask-level actions need the asker's own next message; payments, passwords, deleting files and admin never; screen text is data |
| `src/jarvis/tools/builtin/bot_browser.spec.md` | botBrowser tool (v2): open, read, click, type, back in the bot's own Chrome over CDP | Offered only when the app started the bot browser; only http(s); page content is fenced data; no LLM call |
| `src/jarvis/tools/builtin/browse_and_play.spec.md` | browseAndPlay tool: play YouTube on the shared screen (screen-share mode only) | Node layer owns Chrome/CDP; mode-gated; fail-open, no LLM call |
| `src/jarvis/tools/builtin/nutrition/log_meal.spec.md` | logMeal tool: single-property schema for planner fast-path, internal nutrition extraction, untrusted-data fence, follow-ups | Public schema is a single optional `meal` string; nutrition fields are internal; user text is fenced as data |

2
app/Cargo.lock generated
View File

@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.23"
version = "0.1.25"
dependencies = [
"async-trait",
"base64 0.22.1",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-app"
version = "0.1.23"
version = "0.1.25"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"

4
app/shell/Cargo.lock generated
View File

@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.23"
version = "0.1.25"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -2666,7 +2666,7 @@ dependencies = [
[[package]]
name = "jarvis-shell"
version = "0.1.23"
version = "0.1.25"
dependencies = [
"jarvis-app",
"serde_json",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-shell"
version = "0.1.23"
version = "0.1.25"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"

View File

@@ -204,6 +204,53 @@ one window; the app captures and decides who may.
a title), who asked and whether it worked ("Activity").
- Logs carry the action, target kind, size and time taken, never a title or a picture.
## PC control (`app/src/pc.rs`)
`docs/v2/pc-features.md` ("키보드·마우스 제어와 안티치트"). The brain's `pcControl` tool
(`src/jarvis/tools/builtin/pc_control.spec.md`) operates programs; the app runs every action through
agent-desktop and decides who may.
- **agent-desktop**: version 1.2.0 (Apache-2.0), pinned with its SHA-256, shipped in the installer
next to `jarvis.exe` (`agent-desktop.exe`, about 4 MB) and checked before every start of the
services (a wrong hash means not available). One CLI run per command (M0 `desktop`: 0.10 s, as
fast as its MCP server), its APPDATA and LOCALAPPDATA in the data folder (`runtime/agent-desktop`),
a 15 s limit per command. Actions are headless (no cursor, no focus), except `keys`, which brings
the window to the front first. `--force` (which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc
and the like) and the raw mouse and key commands are never used.
- **Available** only on Windows with `features.pcControl` on, someone in `pcControl.allowedUsers`
and agent-desktop present; otherwise `pcControl` is in the brain's `disabled_tools`. Settings
apply on the next start.
- **Who and what**: every `pc_action` (`voice_worker.spec.md`, "PC control") is decided for its
asker by `permissions::decide` with the action's kind (`windows`, `read`: View; `click`, `close`:
Click; `type`, `keys`: Type; `run`: Run). Refused: people not registered ("PC 제어 화면에 등록된
사람만 PC를 조작할 수 있습니다"), a kind at `never` ("PC 제어 화면에서 이 동작이 꺼져 있습니다").
At `ask` the action is done only with `consent: true`, else the answer is `ask: true` with the
kinds and the window (the brain asks the person; the app does not keep consent).
- **Never**: typing into a password field (agent-desktop reports it, or Windows UI Automation's
IsPassword) is refused ("비밀번호 칸에는 입력하지 않습니다"); a window run as administrator is
refused before acting ("관리자 권한으로 실행된 창은 조작할 수 없습니다"), and `run` never asks for
administrator rights.
- **Anti-cheat**: while a game with an anti-cheat runs, every action but `windows` and `read` is
refused with "<안티치트>가 적용된 게임(<게임>)이 실행 중이어서 키보드/마우스 제어를 할 수
없습니다". Found by its processes, checked at each action: Vanguard by Riot's game processes
(VALORANT, League of Legends; the driver is always loaded), Easy Anti-Cheat, BattlEye, FACEIT,
nProtect GameGuard and XIGNCODE3 by their own processes (list in `pc.rs`, grown by updates), and
`pcControl.antiCheatGames` (program file names the person added on the PC 제어 screen; no switch
turns the check off). The game's name is the game detection's when it runs, else the program's.
- **Emergency stop**: a global hotkey (`pcControl.stopHotkey`, default `Ctrl+Shift+F12`) stops at
once: the running agent-desktop command is killed and every action but `windows` and `read` is
refused ("비상 정지 중입니다. 단축키를 한 번 더 누르거나 PC 제어 화면에서 풀어 주세요") until the
hotkey is pressed again or the PC 제어 screen lifts it. The dashboard shows "PC 제어: 비상 정지";
an app restart lifts it.
- **Results**: `read` and every action answer with the window `{id, program, title}` and its
interactive elements read afresh after the action (at most 200), so the brain says it was done
only from what it sees. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
- **Activity**: each action but `windows` and `read` is `pc.<action>` with the program's file name
(never a title, element name or text), who asked and whether it worked ("Activity").
- Logs carry the action, the result code and the time taken; never a title, element name, value or
typed text.
## Personal notes (`app/src/notes.rs`)
`docs/v2/proactive.md` (제안 4). Each person's standing instructions for how Jarvis answers them,
@@ -662,7 +709,7 @@ start it first, so nothing answers before the brain can, and stop it last.
counted as other programs' memory; when that stop is not confirmed, nothing is read or kept and
the model starts with `expert-vram=0`. Without a card nothing is set. The game
placement (used by the switching that comes next) takes the reserve when it fits beside more
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from.
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from, and once the model answers how much of the card is left free (`memory.free`, without what the driver reserves: `card after the start: N MiB free of T MiB`).
- **A server the app does not start**: with `llm.url` (an OpenAI-compatible `/v1` address, in the
settings file only) the app starts nothing; that server must list a model within 30 seconds.
- **Settings**: `llm.context` (default 32768 tokens per request, 4096-262144) and `llm.port`

View File

@@ -514,7 +514,7 @@ mod perf_tests {
let llm = Arc::new(Llm::with_exe(l.clone(), exe).with_gpu(Arc::new(move || {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
Some(crate::strata::Gpu { id: "GPU-test".into(), used_mib: o2.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total_mib: 24576 })
Some(crate::strata::tests::gpu("GPU-test", o2.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, 24576))
})));
let s = crate::settings::parse_settings(&serde_json::json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();

View File

@@ -116,6 +116,8 @@ pub struct Gpu {
pub id: String,
pub used_mib: i64,
pub total_mib: i64,
/// What is left for programs (`memory.free`): the total less what is used and what the driver reserves.
pub free_mib: i64,
}
/// Reads the card (`nvidia-smi`); None without one.
@@ -123,7 +125,7 @@ pub type GpuReader = Arc<dyn Fn() -> Option<Gpu> + Send + Sync>;
fn nvidia_memory() -> Option<Gpu> {
let mut cmd = std::process::Command::new("nvidia-smi");
cmd.args(["--query-gpu=uuid,memory.used,memory.total", "--format=csv,noheader,nounits"]);
cmd.args(["--query-gpu=uuid,memory.used,memory.total,memory.free", "--format=csv,noheader,nounits"]);
#[cfg(windows)]
{
use std::os::windows::process::CommandExt;
@@ -132,7 +134,8 @@ fn nvidia_memory() -> Option<Gpu> {
let out = cmd.output().ok()?;
let line = String::from_utf8_lossy(&out.stdout).lines().next()?.to_owned();
let mut f = line.split(',').map(str::trim);
Some(Gpu { id: f.next()?.to_owned(), used_mib: f.next()?.parse().ok()?, total_mib: f.next()?.parse().ok()? })
Some(Gpu { id: f.next()?.to_owned(), used_mib: f.next()?.parse().ok()?, total_mib: f.next()?.parse().ok()?,
free_mib: f.next()?.parse().ok()? })
}
/// The newest engine log under Strata's folder, its end only (the placement lines of the last start).
@@ -506,6 +509,11 @@ impl Llm {
Ok(None) => {}
Ok(Some((base, model))) => {
crate::info_log!("llm", "ready: {base} model={model}");
// What the card has left for everything else once the model is in (the voice
// worker's models, a game): a slow answer later can be matched to it.
if let Some(g) = self.read_gpu().await {
crate::info_log!("llm", "card after the start: {} MiB free of {} MiB", g.free_mib, g.total_mib);
}
self.set(LlmState::Ready, None);
}
Err(e) if e == "cancelled" => self.set(LlmState::Off, None),
@@ -651,6 +659,11 @@ pub(crate) mod tests {
/// Other programs on the fake cards: the game PC's 1460 MiB in M0 less the TTS's share, which
/// the app keeps beside and M0 did not run, so M0's placements come out.
/// A card reading as `nvidia-smi` gives it, with nothing reserved by the driver.
pub(crate) fn gpu(id: &str, used_mib: i64, total_mib: i64) -> Gpu {
Gpu { id: id.into(), used_mib, total_mib, free_mib: total_mib - used_mib }
}
pub(crate) const OTHERS: i64 = 1460 - crate::placement::TTS_MIB;
pub(crate) fn layout(tag: &str) -> Layout {
@@ -908,6 +921,26 @@ pub(crate) mod tests {
assert!(launches[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[0]);
}
#[tokio::test]
async fn the_llm_log_tells_how_much_of_the_card_is_free_once_the_model_answers() {
crate::logs::install(crate::logs::LogHub::new(None));
let l = layout("llm-free");
let port = server_after_launch(&l, "list").await;
// The driver keeps some of the card for itself (`memory.reserved`): it is neither used nor free.
let inner = card(&l, "GPU-test", 0);
let reserved = 444;
let reader: GpuReader = Arc::new(move || inner().map(|g| Gpu { free_mib: g.free_mib - reserved, ..g }));
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(reader);
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
let free = 16303 - OTHERS - 9874 - reserved;
let lines = crate::logs::installed().unwrap()
.query(Some("llm"), crate::logs::Level::Info, Some("free"), None, 1000);
assert!(lines.iter().any(|x| x.text.contains(&format!("{free} MiB free of 16303 MiB"))),
"{:?}", lines.iter().map(|x| &x.text).collect::<Vec<_>>());
llm.stop().await.unwrap();
}
/// A fake card for `fake_placed`: 16303 MiB, `OTHERS` held by other programs (and by any
/// server still left over, `leftover` MiB, until a `launch --stop`), and 9874 MiB more while a
/// Strata server runs (the calls file shows the last launch and stop).
@@ -919,7 +952,7 @@ pub(crate) mod tests {
let stopped = c.contains("launch --stop");
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
let used = OTHERS + if running { 9874 } else if stopped { 0 } else { leftover };
Some(Gpu { id: id.into(), used_mib: used, total_mib: 16303 })
Some(gpu(id, used, 16303))
})
}
@@ -953,7 +986,7 @@ pub(crate) mod tests {
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
let r2 = reads.clone();
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: OTHERS, total_mib: 16303 })
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| gpu("GPU-test", OTHERS, 16303))
}));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
@@ -1059,7 +1092,7 @@ pub(crate) mod tests {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
let used = others.load(Ordering::SeqCst) + worker + if running { 9874 } else { 0 };
Some(Gpu { id: "GPU-test".into(), used_mib: used, total_mib: 16303 })
Some(gpu("GPU-test", used, 16303))
})
}
@@ -1117,7 +1150,7 @@ pub(crate) mod tests {
Arc::new(move || {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
Some(Gpu { id: "GPU-test".into(), used_mib: others.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total_mib: total })
Some(gpu("GPU-test", others.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total))
})
}
@@ -1170,7 +1203,7 @@ pub(crate) mod tests {
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
let r2 = reads.clone();
let llm = llm.with_gpu(Arc::new(move || {
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: OTHERS, total_mib: 16303 })
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| gpu("GPU-test", OTHERS, 16303))
}));
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Gaming);
assert!(launches(&l).last().unwrap().contains("--set expert-vram=0 "), "{:?}", launches(&l));
@@ -1187,7 +1220,7 @@ pub(crate) mod tests {
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
let c = std::fs::read_to_string(&calls).unwrap_or_default();
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
Some(Gpu { id: "GPU-small".into(), used_mib: OTHERS + if running { 9874 } else { 0 }, total_mib: 12288 })
Some(gpu("GPU-small", OTHERS + if running { 9874 } else { 0 }, 12288))
}));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
llm.start(&s).await.unwrap();

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,78 @@
# pcControl tool spec (v2 M7)
Lets Jarvis operate programs on this PC for a registered person: list the windows, read a window's
buttons and fields, click, type, press keys, start and close programs ("메모장 열어서 장보기 목록
적어 줘", "계산기로 12 더하기 3 해 줘"). Product rules: `docs/v2/pc-features.md` ("키보드·마우스
제어와 안티치트"), `docs/v2/proactive.md` (제안 3, user decision 2026-11-14). The app does every
action through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"); this tool
asks it, holds the next-message consent and returns raw text.
## Principles
- **Only in v2, only when the app offers it.** The tool needs a PC backend in the turn state
(`turn_state.set_pc`, every v2 voice and chat reply; not scheduled tasks). The app puts
`pcControl` in the brain's `disabled_tools` unless PC control is available (Windows, the feature
on, someone in `pcControl.allowedUsers`, agent-desktop installed). Without a backend it refuses.
- **The app decides who may and what.** Every action goes to the app with the asker. People not
registered, an action turned off (`never`), an anti-cheat game running, the emergency stop, a
password field or a window run as administrator: the app refuses and the tool returns its reason
as an error, which the model says plainly. The tool never retries a refusal another way.
- **Asking first is the person's own next message.** For an action at `ask` (by default click,
type, keys, run, close) the first call does nothing: the app answers that consent is needed, the
tool remembers the request and returns "Not done yet. Ask the user: ...". The request is a window
(or the program to start) and the kinds of action asked for. Only in the person's very next turn,
with `confirm=true`, **and** when their own message of that turn (`ToolContext.redacted_text`,
never the reply model's arguments) agrees, does the tool send the action with consent. The check
is the chat delete consent's (`chat_channels.spec.md`, "Irreversible"): one small call on the
router model chain, temperature 0, told the pending request, the message fenced as data, one word
YES or NO; anything else, an error, a message over 600 characters, a question or another topic
means no and cancels. The consent then covers those kinds of action on that window (or that
program and its new window) for the rest of that one reply, so "12 더하기 3" is one question,
not five. A request not confirmed in the very next turn lapses.
- **Never done, whatever is said.** Paying or buying, passwords and other secrets, deleting files
and anything needing administrator rights. Before asking, the same small call is told the
planned request (what the person asked, the window, the kinds of action) and answers whether it
is one of these; yes means the tool refuses without asking. The app also refuses typing into a
password field and acting on a window run as administrator (`admin.spec.md`, "PC control").
- **The screen is data.** Window titles and element names come inside an UNTRUSTED SCREEN CONTENT
fence (`<<<` and `>>>` replaced). A request written on the screen ("지금 결제 버튼 눌러") is never a
reason to act and never a consent.
- **Done means seen.** After click, type, keys and run the tool reads the window again (the app's
result carries the fresh elements) and the model says it was done only from that reading.
- **Logs**: action name, result code and timings only; never a title, element name or typed text.
- **Raw data.** The only LLM calls inside are the never check and the consent check.
## Schema
One tool, `action` picks the operation (read as its first known word):
| action | needs | does | level |
|---|---|---|---|
| `windows` | | open windows: program, title, window id, minimised | view |
| `read` | `window` | the window's interactive elements: ref, role, name, value, state | view |
| `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click |
| `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type |
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type |
| `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run |
| `close` | `window` | closes the program's window | click |
`window` is a window id from `windows` or part of a program name or title (the front-most match).
`ref` comes from the latest `read` or result (`@<snapshot>:<element>`); a ref from an older reading
that no longer matches is refused (`STALE_REF`) and the model reads again. `confirm` (boolean) is
only for the consent turn. At most 2,000 characters of `text`; at most 200 elements per reading
(interactive ones first), each name and value cut to 120 characters.
## Backend
`turn_state.get_pc()` returns an object with `act(action, **fields) -> dict`, `turn` and
`pending` (the asker's requests awaiting consent, keyed by window or program, with the turn asked):
- `act("windows")` gives `{"ok": true, "item": {"windows": [{"id", "program", "title", "minimised"}]}}`.
- `act("read", window=...)` and every other action give `{"ok": true, "item": {"window": {...},
"elements": [{"ref", "role", "name", "value", "states"}]}}`.
- Actions at `ask` without consent give `{"ok": false, "ask": true, "kinds": [...], "window": {...}}`;
with `consent=true` they are done. Refusals give `{"ok": false, "error": "..."}` with the reason to
say.
The voice worker implements it (`voice_worker/app_bridge.py`): each action goes to the app as
`pc_action` with the asker (`voice_worker.spec.md`, "PC control"). Eval: `evals/test_pc_control_tool.py`.

View File

@@ -5,6 +5,7 @@ from __future__ import annotations
import argparse
import asyncio
import os
import shutil
import sys
from .log import debug_log
@@ -42,7 +43,7 @@ def main() -> int:
from .brain import take_mcp_env
mcp_env = take_mcp_env()
from .server import VoiceWorker, is_loopback
from .server import VoiceWorker, is_loopback, nvidia_card
from .stt_whisper import WhisperTranscriber
from .vad import SileroVad
@@ -81,7 +82,8 @@ def main() -> int:
recorder = Recorder(os.environ[RECORDINGS_ENV])
async def run() -> None:
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain, recorder=recorder)
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain, recorder=recorder,
card=nvidia_card if shutil.which("nvidia-smi") else None)
server = await worker.serve(args.host, args.port)
debug_log(f"voice worker listening on {args.host}:{args.port} tts={'on' if tts else 'off'} "
f"history={'on' if history else 'off'} brain={'on' if brain else 'off'} "

View File

@@ -15,7 +15,7 @@ from typing import Callable
import numpy as np
from .log import debug_log
from .log import debug_log, notice
from .audio import InputConverter, loudness_equalise
from .brain import Brain, MemorySettings
@@ -44,6 +44,20 @@ def is_loopback(host: str) -> bool:
LLM_WAIT_S = 180.0
def nvidia_card() -> tuple[int, int] | None:
"""The first NVIDIA card's free and total memory in MiB (``nvidia-smi``; free leaves out what the
driver reserves), None when unreadable."""
import subprocess
try:
out = subprocess.run(["nvidia-smi", "--query-gpu=memory.free,memory.total", "--format=csv,noheader,nounits"],
capture_output=True, text=True, timeout=5,
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0)).stdout
free, total = (int(v) for v in out.splitlines()[0].split(","))
return free, total
except Exception:
return None
class _LlmPaused(Exception):
"""The app paused the LLM (a game runs): nothing is answered until it is ready again."""
@@ -53,6 +67,7 @@ MAX_NOTES = 20
MAX_NOTE_CHARS = 300
# How long answer traces stay (history.spec.md, "Traces"); removed when the worker starts.
TRACE_KEEP_DAYS = 30
CARD_EVERY_S = 1.0 # how often the card's free memory is read while an answer is written
class _LiveTrace(list):
@@ -93,7 +108,8 @@ class VoiceWorker:
def __init__(self, token: str, transcriber: Transcriber | None, vad_factory: Callable[[], VadStream],
tts_engine: TtsEngine | None = None, hello_timeout_s: float = 5.0, tick_s: float = 0.05,
history=None, brain: Brain | None = None, chat_timeout_s: float = 15.0,
delivery_timeout_s: float = 30.0, recorder=None):
delivery_timeout_s: float = 30.0, recorder=None,
card: Callable[[], tuple[int, int] | None] | None = None):
if not token:
raise ValueError("a connection token is required")
self.token = token
@@ -105,6 +121,8 @@ class VoiceWorker:
self.history = history # jarvis.memory.history.HistoryStore or None
self.recorder = recorder # recordings.Recorder (voice.record on) or None
self.brain = brain
self.card = card # (free MiB, total MiB) of the graphics card, or None without one
self.card_watches: set[asyncio.Task] = set() # readings still running after their answer
# One reply at a time per lane: voice answers never wait behind another person's chat answer
# (conversation.md: with an "answer everything" chat channel the LLM takes two or more
# requests at once). One person's replies stay one at a time across lanes, so their dialogue
@@ -687,7 +705,7 @@ class Session:
history = HistoryContext()
try:
trace.append(self._records_step(history, notes))
async with self._llm():
async with self._llm(), self._card_watch(f"task {rid}"):
answer = await asyncio.to_thread(w.brain.reply, user, text, history, task=True,
notes=notes or [], trace=trace)
except Exception as e: # the brain must never take the session down
@@ -752,6 +770,46 @@ class Session:
w.llm_active -= 1
self._llm_check_idle()
@contextlib.asynccontextmanager
async def _card_watch(self, label: str):
"""While an answer is written: the card's free memory, read every `CARD_EVERY_S`; the least
seen is logged once it ends (a slow answer can then be matched to a full card). The answer
never waits for a reading: one still running when it ends finishes on its own, then logs."""
w = self.worker
card = w.card
if card is None:
yield
return
seen: list[tuple[int, int]] = []
ended = asyncio.Event()
t0, took = time.monotonic(), [0]
async def watch():
while True:
got = await asyncio.to_thread(card)
if got:
seen.append(got)
if ended.is_set():
break
try:
await asyncio.wait_for(ended.wait(), CARD_EVERY_S)
break
except asyncio.TimeoutError:
pass
if seen:
total = seen[-1][1]
notice(f"{label} card: least {min(f for f, _ in seen)} MiB free of {total} MiB "
f"({len(seen)} readings, {took[0]} ms)", "llm")
task = asyncio.get_running_loop().create_task(watch())
w.card_watches.add(task)
task.add_done_callback(w.card_watches.discard)
try:
yield
finally:
took[0] = int((time.monotonic() - t0) * 1000)
ended.set()
def _on_prepare(self, msg: dict) -> None:
"""A call still being said: the brain gets ready for it. One prepare per user runs at a time;
while it runs only the newest text waits, and one about the same as the last one prepared
@@ -909,7 +967,7 @@ class Session:
screen = AppBridge(self, asyncio.get_running_loop(), "screen", *origin)
trace.append(self._records_step(history, notes))
try:
async with self._llm(rid if speak else None):
async with self._llm(rid if speak else None), self._card_watch(f"answer {rid}"):
w.answering += 1
try:
answer = await asyncio.to_thread(w.brain.reply, user, text, history, chat=chat,

View File

@@ -410,6 +410,23 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
dropped. The pictures stay in that one reply (screen_view.spec.md).
- Messages may be up to 4 MiB (pictures), the WebSocket's limit.
### PC control (M7)
The brain's pcControl tool (`src/jarvis/tools/builtin/pc_control.spec.md`) asks the app, which acts
through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"):
```
<- {"type": "pc_action", "id": "p1", "action": "windows|read|click|type|keys|run|close", "user": "<id>",
"via": "voice|chat", "channel": "...", "window": "...", "ref": "@s1:e8", "text": "...", "keys": "ctrl+s",
"program": "notepad", "consent": true}
-> {"type": "pc_action_result", "id": "p1", "ok": true, "item": {...}} // or ok false with "error", or "ask": true
```
- Every voice and chat reply gets a PC backend for its asker (not scheduled tasks); ids and waiting
are those of chat and schedule actions. The worker keeps each asker's requests awaiting consent
with the turn they were asked in, numbered like the chat delete confirmations ("History and
brain"), so only a message sent after the question counts as the next turn.
### Background collection (M6)
With `JARVIS_COLLECT_PATH` set, the worker collects on its own thread, apart from the voice and the
@@ -514,3 +531,12 @@ cancel, and engine failures. The category names the service, so the app's log sc
in its tab (`app/src/admin.spec.md`, "Logs"): `stt` (utterance ends, STT batches, the Whisper model),
`tts` (sentences, cancels, MeloTTS), `llm` (the brain's replies, prepares and judgements),
`voice_worker` for the rest (connections, sessions, channels).
Without debug logging too, each answer and each scheduled task logs one `llm` line once the LLM is
done with it: the least free graphics memory read while it was written, the card's total, how many
readings and how long it took (`answer <id> card: least N MiB free of T MiB (k readings, ms)`). The
card is read with `nvidia-smi` (`memory.free`, which leaves out what the driver reserves) once a
second while the answer is written, in a thread, so the answer never waits for it: a reading still
running when the answer ends finishes on its own and the line follows it, so a short answer gets its
line too. Without `nvidia-smi` nothing is read or logged. The line carries no text, so a
slow answer can be matched to a full card (the engine log has the same answer's timings).

View File

@@ -1568,3 +1568,78 @@ def test_the_judgement_is_told_who_is_in_the_channel(monkeypatch):
assert "Several people" in unknown and "anything unclear" in unknown, "an app that does not say: as before"
assert "one other person" in pair and "unclear, answer yes" in pair
assert "5 people" in group and "Only 1 of the 5" in group and "anything unclear" in group
@pytest.mark.unit
def test_each_answer_logs_the_least_graphics_memory_free_while_it_was_written(store, monkeypatch, capfd):
"""voice_worker.spec.md, "Debug logging": an answer (and a scheduled task) logs the least free
graphics memory read while the LLM wrote it, without debug logging and without its text, so a
slow answer can be matched to a full card."""
import time as _time
from voice_worker import server as worker_server
monkeypatch.setattr(worker_server, "CARD_EVERY_S", 0.02)
# Free memory as nvidia-smi reports it (the driver's reserved share is not free).
readings = iter([7000, 859, 3000])
last = [3000]
def card():
last[0] = next(readings, last[0])
return last[0], 16303
def slow(user, text):
_time.sleep(0.3)
return f"secret answer for {text}"
async def body(url):
async with connect(url) as ws:
await hello(ws)
await send(ws, type="reply", id="r1", user="100", text="private question", via="chat", speak=False)
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
await send(ws, type="reply", id="t1", user="100", text="private task", via="task")
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text" and m["id"] == "t1")
asyncio.run(with_worker(body, store=store, brain=FakeBrain(slow), card=card))
lines = [x for x in capfd.readouterr().err.splitlines() if x.startswith(f"[{'llm':^10}]") and "free" in x]
reply = [x for x in lines if "answer r1" in x]
task = [x for x in lines if "task t1" in x]
assert len(reply) == 1 and len(task) == 1, lines
assert "least 859 MiB free of 16303 MiB" in reply[0], reply
assert "private" not in "".join(lines) and "secret" not in "".join(lines)
@pytest.mark.unit
def test_without_a_card_reading_an_answer_logs_no_card_line(store, capfd):
async def body(url):
async with connect(url) as ws:
await hello(ws)
await send(ws, type="reply", id="r1", user="100", text="q", via="chat", speak=False)
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
asyncio.run(with_worker(body, store=store, brain=FakeBrain(), card=lambda: None))
assert "MiB free" not in capfd.readouterr().err
@pytest.mark.unit
def test_an_answer_done_before_the_first_card_reading_is_logged_once_the_reading_ends(store, capfd):
"""A short answer is not held for the card, and still gets its line once the reading ends."""
import threading
reading = threading.Event()
def card():
reading.wait(5)
return 2048, 16303
async def body(url):
async with connect(url) as ws:
await hello(ws)
await send(ws, type="reply", id="r1", user="100", text="q", via="chat", speak=False)
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
before = capfd.readouterr().err
reading.set()
await asyncio.sleep(0.3)
return before
before = asyncio.run(with_worker(body, store=store, brain=FakeBrain(), card=card))
after = capfd.readouterr().err
assert "answer r1 card" not in before, "the answer did not wait for the card"
assert "answer r1 card: least 2048 MiB free of 16303 MiB (1 readings" in after, after[-1500:]