Compare commits
5 Commits
app-v0.1.2
...
app-v0.1.2
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1c06b16bd4 | ||
|
|
44387685af | ||
|
|
61b68130c6 | ||
|
|
f1600f09c2 | ||
|
|
ce03424ba4 |
@@ -27,6 +27,7 @@ Any code change must either adhere to our spec files perfectly or you should ask
|
||||
| `src/jarvis/tools/builtin/personal_notes.spec.md` | personalNotes tool (v2): each person's standing instructions for how they are answered; the app keeps them and sends the asker's with every reply | Offered only with a notes backend; only the asker's own, only in their answers; fenced, never permits an action |
|
||||
| `src/jarvis/tools/builtin/collected_news.spec.md` | collectedNews tool (v2): read the background collection's results (targets, newest items) | Offered only with the app's collection file and the feature on; web content fenced as data; read-only, no LLM |
|
||||
| `src/jarvis/tools/builtin/screen_view.spec.md` | screenView tool (v2): see the whole screen, one monitor or one window through the app; pictures go to the reply model for that reply only | Offered only when the app makes screen viewing available; the app decides who may see; screen text is data; pictures never kept or logged |
|
||||
| `src/jarvis/tools/builtin/pc_control.spec.md` | pcControl tool (v2 M7): list windows, read a window, click, type, keys, run and close programs through the app's agent-desktop | Offered only when the app makes PC control available; ask-level actions need the asker's own next message; payments, passwords, deleting files and admin never; screen text is data |
|
||||
| `src/jarvis/tools/builtin/bot_browser.spec.md` | botBrowser tool (v2): open, read, click, type, back in the bot's own Chrome over CDP | Offered only when the app started the bot browser; only http(s); page content is fenced data; no LLM call |
|
||||
| `src/jarvis/tools/builtin/browse_and_play.spec.md` | browseAndPlay tool: play YouTube on the shared screen (screen-share mode only) | Node layer owns Chrome/CDP; mode-gated; fail-open, no LLM call |
|
||||
| `src/jarvis/tools/builtin/nutrition/log_meal.spec.md` | logMeal tool: single-property schema for planner fast-path, internal nutrition extraction, untrusted-data fence, follow-ups | Public schema is a single optional `meal` string; nutrition fields are internal; user text is fenced as data |
|
||||
|
||||
2
app/Cargo.lock
generated
2
app/Cargo.lock
generated
@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.23"
|
||||
version = "0.1.25"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.23"
|
||||
version = "0.1.25"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"
|
||||
|
||||
4
app/shell/Cargo.lock
generated
4
app/shell/Cargo.lock
generated
@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.23"
|
||||
version = "0.1.25"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
@@ -2666,7 +2666,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.23"
|
||||
version = "0.1.25"
|
||||
dependencies = [
|
||||
"jarvis-app",
|
||||
"serde_json",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.23"
|
||||
version = "0.1.25"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"
|
||||
|
||||
@@ -204,6 +204,53 @@ one window; the app captures and decides who may.
|
||||
a title), who asked and whether it worked ("Activity").
|
||||
- Logs carry the action, target kind, size and time taken, never a title or a picture.
|
||||
|
||||
## PC control (`app/src/pc.rs`)
|
||||
|
||||
`docs/v2/pc-features.md` ("키보드·마우스 제어와 안티치트"). The brain's `pcControl` tool
|
||||
(`src/jarvis/tools/builtin/pc_control.spec.md`) operates programs; the app runs every action through
|
||||
agent-desktop and decides who may.
|
||||
|
||||
- **agent-desktop**: version 1.2.0 (Apache-2.0), pinned with its SHA-256, shipped in the installer
|
||||
next to `jarvis.exe` (`agent-desktop.exe`, about 4 MB) and checked before every start of the
|
||||
services (a wrong hash means not available). One CLI run per command (M0 `desktop`: 0.10 s, as
|
||||
fast as its MCP server), its APPDATA and LOCALAPPDATA in the data folder (`runtime/agent-desktop`),
|
||||
a 15 s limit per command. Actions are headless (no cursor, no focus), except `keys`, which brings
|
||||
the window to the front first. `--force` (which lifts its block on Alt+F4, Win+R, Ctrl+Shift+Esc
|
||||
and the like) and the raw mouse and key commands are never used.
|
||||
- **Available** only on Windows with `features.pcControl` on, someone in `pcControl.allowedUsers`
|
||||
and agent-desktop present; otherwise `pcControl` is in the brain's `disabled_tools`. Settings
|
||||
apply on the next start.
|
||||
- **Who and what**: every `pc_action` (`voice_worker.spec.md`, "PC control") is decided for its
|
||||
asker by `permissions::decide` with the action's kind (`windows`, `read`: View; `click`, `close`:
|
||||
Click; `type`, `keys`: Type; `run`: Run). Refused: people not registered ("PC 제어 화면에 등록된
|
||||
사람만 PC를 조작할 수 있습니다"), a kind at `never` ("PC 제어 화면에서 이 동작이 꺼져 있습니다").
|
||||
At `ask` the action is done only with `consent: true`, else the answer is `ask: true` with the
|
||||
kinds and the window (the brain asks the person; the app does not keep consent).
|
||||
- **Never**: typing into a password field (agent-desktop reports it, or Windows UI Automation's
|
||||
IsPassword) is refused ("비밀번호 칸에는 입력하지 않습니다"); a window run as administrator is
|
||||
refused before acting ("관리자 권한으로 실행된 창은 조작할 수 없습니다"), and `run` never asks for
|
||||
administrator rights.
|
||||
- **Anti-cheat**: while a game with an anti-cheat runs, every action but `windows` and `read` is
|
||||
refused with "<안티치트>가 적용된 게임(<게임>)이 실행 중이어서 키보드/마우스 제어를 할 수
|
||||
없습니다". Found by its processes, checked at each action: Vanguard by Riot's game processes
|
||||
(VALORANT, League of Legends; the driver is always loaded), Easy Anti-Cheat, BattlEye, FACEIT,
|
||||
nProtect GameGuard and XIGNCODE3 by their own processes (list in `pc.rs`, grown by updates), and
|
||||
`pcControl.antiCheatGames` (program file names the person added on the PC 제어 screen; no switch
|
||||
turns the check off). The game's name is the game detection's when it runs, else the program's.
|
||||
- **Emergency stop**: a global hotkey (`pcControl.stopHotkey`, default `Ctrl+Shift+F12`) stops at
|
||||
once: the running agent-desktop command is killed and every action but `windows` and `read` is
|
||||
refused ("비상 정지 중입니다. 단축키를 한 번 더 누르거나 PC 제어 화면에서 풀어 주세요") until the
|
||||
hotkey is pressed again or the PC 제어 screen lifts it. The dashboard shows "PC 제어: 비상 정지";
|
||||
an app restart lifts it.
|
||||
- **Results**: `read` and every action answer with the window `{id, program, title}` and its
|
||||
interactive elements read afresh after the action (at most 200), so the brain says it was done
|
||||
only from what it sees. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
|
||||
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
|
||||
- **Activity**: each action but `windows` and `read` is `pc.<action>` with the program's file name
|
||||
(never a title, element name or text), who asked and whether it worked ("Activity").
|
||||
- Logs carry the action, the result code and the time taken; never a title, element name, value or
|
||||
typed text.
|
||||
|
||||
## Personal notes (`app/src/notes.rs`)
|
||||
|
||||
`docs/v2/proactive.md` (제안 4). Each person's standing instructions for how Jarvis answers them,
|
||||
@@ -662,7 +709,7 @@ start it first, so nothing answers before the brain can, and stop it last.
|
||||
counted as other programs' memory; when that stop is not confirmed, nothing is read or kept and
|
||||
the model starts with `expert-vram=0`. Without a card nothing is set. The game
|
||||
placement (used by the switching that comes next) takes the reserve when it fits beside more
|
||||
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from.
|
||||
than 2 GiB of experts, else 2 GiB and the room that leaves, else 0 GiB and its room, else pause. The `llm` log tab gets the placement and what it came from, and once the model answers how much of the card is left free (`memory.free`, without what the driver reserves: `card after the start: N MiB free of T MiB`).
|
||||
- **A server the app does not start**: with `llm.url` (an OpenAI-compatible `/v1` address, in the
|
||||
settings file only) the app starts nothing; that server must list a model within 30 seconds.
|
||||
- **Settings**: `llm.context` (default 32768 tokens per request, 4096-262144) and `llm.port`
|
||||
|
||||
@@ -514,7 +514,7 @@ mod perf_tests {
|
||||
let llm = Arc::new(Llm::with_exe(l.clone(), exe).with_gpu(Arc::new(move || {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
Some(crate::strata::Gpu { id: "GPU-test".into(), used_mib: o2.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total_mib: 24576 })
|
||||
Some(crate::strata::tests::gpu("GPU-test", o2.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, 24576))
|
||||
})));
|
||||
let s = crate::settings::parse_settings(&serde_json::json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
|
||||
@@ -116,6 +116,8 @@ pub struct Gpu {
|
||||
pub id: String,
|
||||
pub used_mib: i64,
|
||||
pub total_mib: i64,
|
||||
/// What is left for programs (`memory.free`): the total less what is used and what the driver reserves.
|
||||
pub free_mib: i64,
|
||||
}
|
||||
|
||||
/// Reads the card (`nvidia-smi`); None without one.
|
||||
@@ -123,7 +125,7 @@ pub type GpuReader = Arc<dyn Fn() -> Option<Gpu> + Send + Sync>;
|
||||
|
||||
fn nvidia_memory() -> Option<Gpu> {
|
||||
let mut cmd = std::process::Command::new("nvidia-smi");
|
||||
cmd.args(["--query-gpu=uuid,memory.used,memory.total", "--format=csv,noheader,nounits"]);
|
||||
cmd.args(["--query-gpu=uuid,memory.used,memory.total,memory.free", "--format=csv,noheader,nounits"]);
|
||||
#[cfg(windows)]
|
||||
{
|
||||
use std::os::windows::process::CommandExt;
|
||||
@@ -132,7 +134,8 @@ fn nvidia_memory() -> Option<Gpu> {
|
||||
let out = cmd.output().ok()?;
|
||||
let line = String::from_utf8_lossy(&out.stdout).lines().next()?.to_owned();
|
||||
let mut f = line.split(',').map(str::trim);
|
||||
Some(Gpu { id: f.next()?.to_owned(), used_mib: f.next()?.parse().ok()?, total_mib: f.next()?.parse().ok()? })
|
||||
Some(Gpu { id: f.next()?.to_owned(), used_mib: f.next()?.parse().ok()?, total_mib: f.next()?.parse().ok()?,
|
||||
free_mib: f.next()?.parse().ok()? })
|
||||
}
|
||||
|
||||
/// The newest engine log under Strata's folder, its end only (the placement lines of the last start).
|
||||
@@ -506,6 +509,11 @@ impl Llm {
|
||||
Ok(None) => {}
|
||||
Ok(Some((base, model))) => {
|
||||
crate::info_log!("llm", "ready: {base} model={model}");
|
||||
// What the card has left for everything else once the model is in (the voice
|
||||
// worker's models, a game): a slow answer later can be matched to it.
|
||||
if let Some(g) = self.read_gpu().await {
|
||||
crate::info_log!("llm", "card after the start: {} MiB free of {} MiB", g.free_mib, g.total_mib);
|
||||
}
|
||||
self.set(LlmState::Ready, None);
|
||||
}
|
||||
Err(e) if e == "cancelled" => self.set(LlmState::Off, None),
|
||||
@@ -651,6 +659,11 @@ pub(crate) mod tests {
|
||||
|
||||
/// Other programs on the fake cards: the game PC's 1460 MiB in M0 less the TTS's share, which
|
||||
/// the app keeps beside and M0 did not run, so M0's placements come out.
|
||||
/// A card reading as `nvidia-smi` gives it, with nothing reserved by the driver.
|
||||
pub(crate) fn gpu(id: &str, used_mib: i64, total_mib: i64) -> Gpu {
|
||||
Gpu { id: id.into(), used_mib, total_mib, free_mib: total_mib - used_mib }
|
||||
}
|
||||
|
||||
pub(crate) const OTHERS: i64 = 1460 - crate::placement::TTS_MIB;
|
||||
|
||||
pub(crate) fn layout(tag: &str) -> Layout {
|
||||
@@ -908,6 +921,26 @@ pub(crate) mod tests {
|
||||
assert!(launches[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[0]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_llm_log_tells_how_much_of_the_card_is_free_once_the_model_answers() {
|
||||
crate::logs::install(crate::logs::LogHub::new(None));
|
||||
let l = layout("llm-free");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
// The driver keeps some of the card for itself (`memory.reserved`): it is neither used nor free.
|
||||
let inner = card(&l, "GPU-test", 0);
|
||||
let reserved = 444;
|
||||
let reader: GpuReader = Arc::new(move || inner().map(|g| Gpu { free_mib: g.free_mib - reserved, ..g }));
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(reader);
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
|
||||
let free = 16303 - OTHERS - 9874 - reserved;
|
||||
let lines = crate::logs::installed().unwrap()
|
||||
.query(Some("llm"), crate::logs::Level::Info, Some("free"), None, 1000);
|
||||
assert!(lines.iter().any(|x| x.text.contains(&format!("{free} MiB free of 16303 MiB"))),
|
||||
"{:?}", lines.iter().map(|x| &x.text).collect::<Vec<_>>());
|
||||
llm.stop().await.unwrap();
|
||||
}
|
||||
|
||||
/// A fake card for `fake_placed`: 16303 MiB, `OTHERS` held by other programs (and by any
|
||||
/// server still left over, `leftover` MiB, until a `launch --stop`), and 9874 MiB more while a
|
||||
/// Strata server runs (the calls file shows the last launch and stop).
|
||||
@@ -919,7 +952,7 @@ pub(crate) mod tests {
|
||||
let stopped = c.contains("launch --stop");
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
let used = OTHERS + if running { 9874 } else if stopped { 0 } else { leftover };
|
||||
Some(Gpu { id: id.into(), used_mib: used, total_mib: 16303 })
|
||||
Some(gpu(id, used, 16303))
|
||||
})
|
||||
}
|
||||
|
||||
@@ -953,7 +986,7 @@ pub(crate) mod tests {
|
||||
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let r2 = reads.clone();
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: OTHERS, total_mib: 16303 })
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| gpu("GPU-test", OTHERS, 16303))
|
||||
}));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
|
||||
@@ -1059,7 +1092,7 @@ pub(crate) mod tests {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
let used = others.load(Ordering::SeqCst) + worker + if running { 9874 } else { 0 };
|
||||
Some(Gpu { id: "GPU-test".into(), used_mib: used, total_mib: 16303 })
|
||||
Some(gpu("GPU-test", used, 16303))
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1117,7 +1150,7 @@ pub(crate) mod tests {
|
||||
Arc::new(move || {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
Some(Gpu { id: "GPU-test".into(), used_mib: others.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total_mib: total })
|
||||
Some(gpu("GPU-test", others.load(Ordering::SeqCst) + if running { 9874 } else { 0 }, total))
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1170,7 +1203,7 @@ pub(crate) mod tests {
|
||||
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let r2 = reads.clone();
|
||||
let llm = llm.with_gpu(Arc::new(move || {
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| Gpu { id: "GPU-test".into(), used_mib: OTHERS, total_mib: 16303 })
|
||||
(r2.fetch_add(1, Ordering::SeqCst) == 0).then(|| gpu("GPU-test", OTHERS, 16303))
|
||||
}));
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming, 0, true).await.unwrap(), Stage::Gaming);
|
||||
assert!(launches(&l).last().unwrap().contains("--set expert-vram=0 "), "{:?}", launches(&l));
|
||||
@@ -1187,7 +1220,7 @@ pub(crate) mod tests {
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
Some(Gpu { id: "GPU-small".into(), used_mib: OTHERS + if running { 9874 } else { 0 }, total_mib: 12288 })
|
||||
Some(gpu("GPU-small", OTHERS + if running { 9874 } else { 0 }, 12288))
|
||||
}));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
|
||||
File diff suppressed because one or more lines are too long
78
src/jarvis/tools/builtin/pc_control.spec.md
Normal file
78
src/jarvis/tools/builtin/pc_control.spec.md
Normal file
@@ -0,0 +1,78 @@
|
||||
# pcControl tool spec (v2 M7)
|
||||
|
||||
Lets Jarvis operate programs on this PC for a registered person: list the windows, read a window's
|
||||
buttons and fields, click, type, press keys, start and close programs ("메모장 열어서 장보기 목록
|
||||
적어 줘", "계산기로 12 더하기 3 해 줘"). Product rules: `docs/v2/pc-features.md` ("키보드·마우스
|
||||
제어와 안티치트"), `docs/v2/proactive.md` (제안 3, user decision 2026-11-14). The app does every
|
||||
action through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"); this tool
|
||||
asks it, holds the next-message consent and returns raw text.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Only in v2, only when the app offers it.** The tool needs a PC backend in the turn state
|
||||
(`turn_state.set_pc`, every v2 voice and chat reply; not scheduled tasks). The app puts
|
||||
`pcControl` in the brain's `disabled_tools` unless PC control is available (Windows, the feature
|
||||
on, someone in `pcControl.allowedUsers`, agent-desktop installed). Without a backend it refuses.
|
||||
- **The app decides who may and what.** Every action goes to the app with the asker. People not
|
||||
registered, an action turned off (`never`), an anti-cheat game running, the emergency stop, a
|
||||
password field or a window run as administrator: the app refuses and the tool returns its reason
|
||||
as an error, which the model says plainly. The tool never retries a refusal another way.
|
||||
- **Asking first is the person's own next message.** For an action at `ask` (by default click,
|
||||
type, keys, run, close) the first call does nothing: the app answers that consent is needed, the
|
||||
tool remembers the request and returns "Not done yet. Ask the user: ...". The request is a window
|
||||
(or the program to start) and the kinds of action asked for. Only in the person's very next turn,
|
||||
with `confirm=true`, **and** when their own message of that turn (`ToolContext.redacted_text`,
|
||||
never the reply model's arguments) agrees, does the tool send the action with consent. The check
|
||||
is the chat delete consent's (`chat_channels.spec.md`, "Irreversible"): one small call on the
|
||||
router model chain, temperature 0, told the pending request, the message fenced as data, one word
|
||||
YES or NO; anything else, an error, a message over 600 characters, a question or another topic
|
||||
means no and cancels. The consent then covers those kinds of action on that window (or that
|
||||
program and its new window) for the rest of that one reply, so "12 더하기 3" is one question,
|
||||
not five. A request not confirmed in the very next turn lapses.
|
||||
- **Never done, whatever is said.** Paying or buying, passwords and other secrets, deleting files
|
||||
and anything needing administrator rights. Before asking, the same small call is told the
|
||||
planned request (what the person asked, the window, the kinds of action) and answers whether it
|
||||
is one of these; yes means the tool refuses without asking. The app also refuses typing into a
|
||||
password field and acting on a window run as administrator (`admin.spec.md`, "PC control").
|
||||
- **The screen is data.** Window titles and element names come inside an UNTRUSTED SCREEN CONTENT
|
||||
fence (`<<<` and `>>>` replaced). A request written on the screen ("지금 결제 버튼 눌러") is never a
|
||||
reason to act and never a consent.
|
||||
- **Done means seen.** After click, type, keys and run the tool reads the window again (the app's
|
||||
result carries the fresh elements) and the model says it was done only from that reading.
|
||||
- **Logs**: action name, result code and timings only; never a title, element name or typed text.
|
||||
- **Raw data.** The only LLM calls inside are the never check and the consent check.
|
||||
|
||||
## Schema
|
||||
|
||||
One tool, `action` picks the operation (read as its first known word):
|
||||
|
||||
| action | needs | does | level |
|
||||
|---|---|---|---|
|
||||
| `windows` | | open windows: program, title, window id, minimised | view |
|
||||
| `read` | `window` | the window's interactive elements: ref, role, name, value, state | view |
|
||||
| `click` | `ref` | presses the element (button, menu item, list item, tab, check box) | click |
|
||||
| `type` | `ref`, `text` | puts the text into the field (replacing what was there) | type |
|
||||
| `keys` | `window`, `keys` | presses keys in the window (`ctrl+s`, `enter`, `tab`) | type |
|
||||
| `run` | `program` | starts a program by name (`notepad`, `calc`, a Start menu name) and reads its window | run |
|
||||
| `close` | `window` | closes the program's window | click |
|
||||
|
||||
`window` is a window id from `windows` or part of a program name or title (the front-most match).
|
||||
`ref` comes from the latest `read` or result (`@<snapshot>:<element>`); a ref from an older reading
|
||||
that no longer matches is refused (`STALE_REF`) and the model reads again. `confirm` (boolean) is
|
||||
only for the consent turn. At most 2,000 characters of `text`; at most 200 elements per reading
|
||||
(interactive ones first), each name and value cut to 120 characters.
|
||||
|
||||
## Backend
|
||||
|
||||
`turn_state.get_pc()` returns an object with `act(action, **fields) -> dict`, `turn` and
|
||||
`pending` (the asker's requests awaiting consent, keyed by window or program, with the turn asked):
|
||||
|
||||
- `act("windows")` gives `{"ok": true, "item": {"windows": [{"id", "program", "title", "minimised"}]}}`.
|
||||
- `act("read", window=...)` and every other action give `{"ok": true, "item": {"window": {...},
|
||||
"elements": [{"ref", "role", "name", "value", "states"}]}}`.
|
||||
- Actions at `ask` without consent give `{"ok": false, "ask": true, "kinds": [...], "window": {...}}`;
|
||||
with `consent=true` they are done. Refusals give `{"ok": false, "error": "..."}` with the reason to
|
||||
say.
|
||||
|
||||
The voice worker implements it (`voice_worker/app_bridge.py`): each action goes to the app as
|
||||
`pc_action` with the asker (`voice_worker.spec.md`, "PC control"). Eval: `evals/test_pc_control_tool.py`.
|
||||
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
from .log import debug_log
|
||||
@@ -42,7 +43,7 @@ def main() -> int:
|
||||
from .brain import take_mcp_env
|
||||
mcp_env = take_mcp_env()
|
||||
|
||||
from .server import VoiceWorker, is_loopback
|
||||
from .server import VoiceWorker, is_loopback, nvidia_card
|
||||
from .stt_whisper import WhisperTranscriber
|
||||
from .vad import SileroVad
|
||||
|
||||
@@ -81,7 +82,8 @@ def main() -> int:
|
||||
recorder = Recorder(os.environ[RECORDINGS_ENV])
|
||||
|
||||
async def run() -> None:
|
||||
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain, recorder=recorder)
|
||||
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain, recorder=recorder,
|
||||
card=nvidia_card if shutil.which("nvidia-smi") else None)
|
||||
server = await worker.serve(args.host, args.port)
|
||||
debug_log(f"voice worker listening on {args.host}:{args.port} tts={'on' if tts else 'off'} "
|
||||
f"history={'on' if history else 'off'} brain={'on' if brain else 'off'} "
|
||||
|
||||
@@ -15,7 +15,7 @@ from typing import Callable
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .log import debug_log
|
||||
from .log import debug_log, notice
|
||||
|
||||
from .audio import InputConverter, loudness_equalise
|
||||
from .brain import Brain, MemorySettings
|
||||
@@ -44,6 +44,20 @@ def is_loopback(host: str) -> bool:
|
||||
LLM_WAIT_S = 180.0
|
||||
|
||||
|
||||
def nvidia_card() -> tuple[int, int] | None:
|
||||
"""The first NVIDIA card's free and total memory in MiB (``nvidia-smi``; free leaves out what the
|
||||
driver reserves), None when unreadable."""
|
||||
import subprocess
|
||||
try:
|
||||
out = subprocess.run(["nvidia-smi", "--query-gpu=memory.free,memory.total", "--format=csv,noheader,nounits"],
|
||||
capture_output=True, text=True, timeout=5,
|
||||
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0)).stdout
|
||||
free, total = (int(v) for v in out.splitlines()[0].split(","))
|
||||
return free, total
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
class _LlmPaused(Exception):
|
||||
"""The app paused the LLM (a game runs): nothing is answered until it is ready again."""
|
||||
|
||||
@@ -53,6 +67,7 @@ MAX_NOTES = 20
|
||||
MAX_NOTE_CHARS = 300
|
||||
# How long answer traces stay (history.spec.md, "Traces"); removed when the worker starts.
|
||||
TRACE_KEEP_DAYS = 30
|
||||
CARD_EVERY_S = 1.0 # how often the card's free memory is read while an answer is written
|
||||
|
||||
|
||||
class _LiveTrace(list):
|
||||
@@ -93,7 +108,8 @@ class VoiceWorker:
|
||||
def __init__(self, token: str, transcriber: Transcriber | None, vad_factory: Callable[[], VadStream],
|
||||
tts_engine: TtsEngine | None = None, hello_timeout_s: float = 5.0, tick_s: float = 0.05,
|
||||
history=None, brain: Brain | None = None, chat_timeout_s: float = 15.0,
|
||||
delivery_timeout_s: float = 30.0, recorder=None):
|
||||
delivery_timeout_s: float = 30.0, recorder=None,
|
||||
card: Callable[[], tuple[int, int] | None] | None = None):
|
||||
if not token:
|
||||
raise ValueError("a connection token is required")
|
||||
self.token = token
|
||||
@@ -105,6 +121,8 @@ class VoiceWorker:
|
||||
self.history = history # jarvis.memory.history.HistoryStore or None
|
||||
self.recorder = recorder # recordings.Recorder (voice.record on) or None
|
||||
self.brain = brain
|
||||
self.card = card # (free MiB, total MiB) of the graphics card, or None without one
|
||||
self.card_watches: set[asyncio.Task] = set() # readings still running after their answer
|
||||
# One reply at a time per lane: voice answers never wait behind another person's chat answer
|
||||
# (conversation.md: with an "answer everything" chat channel the LLM takes two or more
|
||||
# requests at once). One person's replies stay one at a time across lanes, so their dialogue
|
||||
@@ -687,7 +705,7 @@ class Session:
|
||||
history = HistoryContext()
|
||||
try:
|
||||
trace.append(self._records_step(history, notes))
|
||||
async with self._llm():
|
||||
async with self._llm(), self._card_watch(f"task {rid}"):
|
||||
answer = await asyncio.to_thread(w.brain.reply, user, text, history, task=True,
|
||||
notes=notes or [], trace=trace)
|
||||
except Exception as e: # the brain must never take the session down
|
||||
@@ -752,6 +770,46 @@ class Session:
|
||||
w.llm_active -= 1
|
||||
self._llm_check_idle()
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _card_watch(self, label: str):
|
||||
"""While an answer is written: the card's free memory, read every `CARD_EVERY_S`; the least
|
||||
seen is logged once it ends (a slow answer can then be matched to a full card). The answer
|
||||
never waits for a reading: one still running when it ends finishes on its own, then logs."""
|
||||
w = self.worker
|
||||
card = w.card
|
||||
if card is None:
|
||||
yield
|
||||
return
|
||||
seen: list[tuple[int, int]] = []
|
||||
ended = asyncio.Event()
|
||||
t0, took = time.monotonic(), [0]
|
||||
|
||||
async def watch():
|
||||
while True:
|
||||
got = await asyncio.to_thread(card)
|
||||
if got:
|
||||
seen.append(got)
|
||||
if ended.is_set():
|
||||
break
|
||||
try:
|
||||
await asyncio.wait_for(ended.wait(), CARD_EVERY_S)
|
||||
break
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
if seen:
|
||||
total = seen[-1][1]
|
||||
notice(f"{label} card: least {min(f for f, _ in seen)} MiB free of {total} MiB "
|
||||
f"({len(seen)} readings, {took[0]} ms)", "llm")
|
||||
|
||||
task = asyncio.get_running_loop().create_task(watch())
|
||||
w.card_watches.add(task)
|
||||
task.add_done_callback(w.card_watches.discard)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
took[0] = int((time.monotonic() - t0) * 1000)
|
||||
ended.set()
|
||||
|
||||
def _on_prepare(self, msg: dict) -> None:
|
||||
"""A call still being said: the brain gets ready for it. One prepare per user runs at a time;
|
||||
while it runs only the newest text waits, and one about the same as the last one prepared
|
||||
@@ -909,7 +967,7 @@ class Session:
|
||||
screen = AppBridge(self, asyncio.get_running_loop(), "screen", *origin)
|
||||
trace.append(self._records_step(history, notes))
|
||||
try:
|
||||
async with self._llm(rid if speak else None):
|
||||
async with self._llm(rid if speak else None), self._card_watch(f"answer {rid}"):
|
||||
w.answering += 1
|
||||
try:
|
||||
answer = await asyncio.to_thread(w.brain.reply, user, text, history, chat=chat,
|
||||
|
||||
@@ -410,6 +410,23 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
|
||||
dropped. The pictures stay in that one reply (screen_view.spec.md).
|
||||
- Messages may be up to 4 MiB (pictures), the WebSocket's limit.
|
||||
|
||||
### PC control (M7)
|
||||
|
||||
The brain's pcControl tool (`src/jarvis/tools/builtin/pc_control.spec.md`) asks the app, which acts
|
||||
through agent-desktop and decides who may (`app/src/admin.spec.md`, "PC control"):
|
||||
|
||||
```
|
||||
<- {"type": "pc_action", "id": "p1", "action": "windows|read|click|type|keys|run|close", "user": "<id>",
|
||||
"via": "voice|chat", "channel": "...", "window": "...", "ref": "@s1:e8", "text": "...", "keys": "ctrl+s",
|
||||
"program": "notepad", "consent": true}
|
||||
-> {"type": "pc_action_result", "id": "p1", "ok": true, "item": {...}} // or ok false with "error", or "ask": true
|
||||
```
|
||||
|
||||
- Every voice and chat reply gets a PC backend for its asker (not scheduled tasks); ids and waiting
|
||||
are those of chat and schedule actions. The worker keeps each asker's requests awaiting consent
|
||||
with the turn they were asked in, numbered like the chat delete confirmations ("History and
|
||||
brain"), so only a message sent after the question counts as the next turn.
|
||||
|
||||
### Background collection (M6)
|
||||
|
||||
With `JARVIS_COLLECT_PATH` set, the worker collects on its own thread, apart from the voice and the
|
||||
@@ -514,3 +531,12 @@ cancel, and engine failures. The category names the service, so the app's log sc
|
||||
in its tab (`app/src/admin.spec.md`, "Logs"): `stt` (utterance ends, STT batches, the Whisper model),
|
||||
`tts` (sentences, cancels, MeloTTS), `llm` (the brain's replies, prepares and judgements),
|
||||
`voice_worker` for the rest (connections, sessions, channels).
|
||||
|
||||
Without debug logging too, each answer and each scheduled task logs one `llm` line once the LLM is
|
||||
done with it: the least free graphics memory read while it was written, the card's total, how many
|
||||
readings and how long it took (`answer <id> card: least N MiB free of T MiB (k readings, ms)`). The
|
||||
card is read with `nvidia-smi` (`memory.free`, which leaves out what the driver reserves) once a
|
||||
second while the answer is written, in a thread, so the answer never waits for it: a reading still
|
||||
running when the answer ends finishes on its own and the line follows it, so a short answer gets its
|
||||
line too. Without `nvidia-smi` nothing is read or logged. The line carries no text, so a
|
||||
slow answer can be matched to a full card (the engine log has the same answer's timings).
|
||||
|
||||
@@ -1568,3 +1568,78 @@ def test_the_judgement_is_told_who_is_in_the_channel(monkeypatch):
|
||||
assert "Several people" in unknown and "anything unclear" in unknown, "an app that does not say: as before"
|
||||
assert "one other person" in pair and "unclear, answer yes" in pair
|
||||
assert "5 people" in group and "Only 1 of the 5" in group and "anything unclear" in group
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_each_answer_logs_the_least_graphics_memory_free_while_it_was_written(store, monkeypatch, capfd):
|
||||
"""voice_worker.spec.md, "Debug logging": an answer (and a scheduled task) logs the least free
|
||||
graphics memory read while the LLM wrote it, without debug logging and without its text, so a
|
||||
slow answer can be matched to a full card."""
|
||||
import time as _time
|
||||
from voice_worker import server as worker_server
|
||||
monkeypatch.setattr(worker_server, "CARD_EVERY_S", 0.02)
|
||||
# Free memory as nvidia-smi reports it (the driver's reserved share is not free).
|
||||
readings = iter([7000, 859, 3000])
|
||||
last = [3000]
|
||||
|
||||
def card():
|
||||
last[0] = next(readings, last[0])
|
||||
return last[0], 16303
|
||||
|
||||
def slow(user, text):
|
||||
_time.sleep(0.3)
|
||||
return f"secret answer for {text}"
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="reply", id="r1", user="100", text="private question", via="chat", speak=False)
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
await send(ws, type="reply", id="t1", user="100", text="private task", via="task")
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text" and m["id"] == "t1")
|
||||
|
||||
asyncio.run(with_worker(body, store=store, brain=FakeBrain(slow), card=card))
|
||||
lines = [x for x in capfd.readouterr().err.splitlines() if x.startswith(f"[{'llm':^10}]") and "free" in x]
|
||||
reply = [x for x in lines if "answer r1" in x]
|
||||
task = [x for x in lines if "task t1" in x]
|
||||
assert len(reply) == 1 and len(task) == 1, lines
|
||||
assert "least 859 MiB free of 16303 MiB" in reply[0], reply
|
||||
assert "private" not in "".join(lines) and "secret" not in "".join(lines)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_without_a_card_reading_an_answer_logs_no_card_line(store, capfd):
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="reply", id="r1", user="100", text="q", via="chat", speak=False)
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
|
||||
asyncio.run(with_worker(body, store=store, brain=FakeBrain(), card=lambda: None))
|
||||
assert "MiB free" not in capfd.readouterr().err
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_an_answer_done_before_the_first_card_reading_is_logged_once_the_reading_ends(store, capfd):
|
||||
"""A short answer is not held for the card, and still gets its line once the reading ends."""
|
||||
import threading
|
||||
reading = threading.Event()
|
||||
|
||||
def card():
|
||||
reading.wait(5)
|
||||
return 2048, 16303
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="reply", id="r1", user="100", text="q", via="chat", speak=False)
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
before = capfd.readouterr().err
|
||||
reading.set()
|
||||
await asyncio.sleep(0.3)
|
||||
return before
|
||||
|
||||
before = asyncio.run(with_worker(body, store=store, brain=FakeBrain(), card=card))
|
||||
after = capfd.readouterr().err
|
||||
assert "answer r1 card" not in before, "the answer did not wait for the card"
|
||||
assert "answer r1 card: least 2048 MiB free of 16303 MiB (1 readings" in after, after[-1500:]
|
||||
|
||||
Reference in New Issue
Block a user