feat(app): place the LLM's experts so the speech-to-text worker keeps its room

- with an NVIDIA card the app sets --expert-vram itself by the M0 kit's formula: what the card
  holds before the start, the model without experts, the STT worker's share (2740 MiB), the
  engine's margin (768 MiB) and room for programs opened later (game.otherRoomGb, default 1 GB)
  are left free; Strata's own placement filled the card and left the STT no room
- the model without experts is measured once per context and card (a start with
  expert-vram=0, one short answer, the card's memory and the engine log's staging), kept in
  config/llm-probe.json; the real start replaces that server
- placement.rs also works out the game placement (the reserve, else the 2 GiB minimum and the
  room a game really gets, else pause) for the switching that comes next
- with the game PC's M0 numbers the normal placement comes out at 1.5 GiB, as M0 measured

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-14 09:53:20 +09:00
parent 7e83b09dd5
commit af6f2c855d
9 changed files with 333 additions and 21 deletions

View File

@@ -335,7 +335,8 @@ elsewhere it never sees a game). It never looks at the game's screen.
- `game` (new section): `detect` (게임 감지, on by default), `games` (programs that are games, as
file names, ".exe" may be left out) and `notGames` (programs that are never games); each list at
most 100 names, no folders. Read when the services start.
most 100 names, no folders; `otherRoomGb` (0-8, default 1), the graphics memory the normal
placement leaves for programs opened later (above, "LLM"). Read when the services start.
- The foreground program is a game, in this order: in `notGames`: no; in `games`: yes
(`registered`); a browser (also the bot's Chrome), video player, the desktop, Discord, a
launcher's own window or this app: no; under a launcher's game folder (the folder after
@@ -377,6 +378,18 @@ start it first, so nothing answers before the brain can, and stop it last.
accepts the start (`Host::prepare_start`), never by the start itself, so a stop between accepting
and running the start cancels it before Strata runs anything. Not installed: the start fails and says to
run the install.
- **Placement** (`app/src/placement.rs`, `docs/v2/pc-features.md` "PC마다 값을 계산"): with an
NVIDIA card (`nvidia-smi`) the app sets the routed experts' graphics memory itself, `--set
expert-vram=<GiB> --set prefill=2048`, as the M0 kit did; Strata's own placement fills the card
and leaves the speech-to-text worker no room, and its qwen36 engine takes no reserve. Experts get
what is left of the card after what it holds before the start (other programs), the model without
experts, the STT worker's share (2740 MiB: its model and ten speakers at once, M0 `stt`), the
engine's margin (768 MiB) and the room for programs opened later (`game.otherRoomGb`, default
1 GB); the budget first holds the long-prompt staging, rounded down to 0.1 GiB, 0 when nothing is
left. What the model holds without experts is measured once per context and card (the first-run
self-check: a start with `expert-vram=0`, one short answer, then the card's memory again and the
staging from the engine log), kept in `config/llm-probe.json`; the real start then replaces that
server. Without a card nothing is set. The `llm` log tab gets the placement and what it came from.
- **A server the app does not start**: with `llm.url` (an OpenAI-compatible `/v1` address, in the
settings file only) the app starts nothing; that server must list a model within 30 seconds.
- **Settings**: `llm.context` (default 32768 tokens per request, 4096-262144) and `llm.port`

View File

@@ -479,7 +479,7 @@ mod tests {
use super::*;
fn settings() -> GameSettings {
GameSettings { detect: true, games: vec![], not_games: vec![] }
GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 }
}
fn fg(path: &str, covers: bool, no_frame: bool, gpu: f64) -> Foreground {

View File

@@ -21,6 +21,7 @@ pub mod frames;
pub mod install;
pub mod logs;
pub mod playback;
pub mod placement;
pub mod policy;
pub mod receive;
pub mod secrets;

126
app/src/placement.rs Normal file
View File

@@ -0,0 +1,126 @@
//! Where the LLM's routed experts go (`docs/v2/pc-features.md`, "PC마다 값을 계산"; admin.spec.md,
//! "LLM placement"): the one model is placed by the M0 kit's formula, so the speech-to-text worker
//! and other programs keep their room on the graphics card. Strata's automatic placement fills the
//! card with experts and left the STT no room (M0 `gamemode_stt`), and its qwen36 engine takes no
//! VRAM reserve, so the app sets `--expert-vram` itself.
use serde::{Deserialize, Serialize};
/// What Strata's qwen36 engine keeps free itself when it places experts (M0 kit).
pub const MARGIN_MIB: i64 = 768;
/// The STT worker's share: its model (1300 MiB) and what ten speakers at once add (1440 MiB),
/// measured on the game PC (M0 `stt`). The services start the LLM before the worker, so both count.
pub const STT_MIB: i64 = 1300 + 1440;
/// The smallest expert budget in a game: it holds the long-prompt staging (about 1.1 GiB) first,
/// so 2 GiB is the least that loads experts, and it was 1.5-2.1 times faster than none in a real
/// game (user decision, 2026-11-10).
pub const MIN_GAME_EXPERT_GIB: f64 = 2.0;
/// The context in a game (16K keeps room for the history, tools and results; 8K saved only 0.4 GB).
pub const GAME_CONTEXT: u32 = 16384;
/// What one start without experts at `context` showed on this PC (the first-run self-check).
#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Probe {
pub context: u32,
/// The card's size when it was measured (another card means measuring again).
pub total_mib: i64,
/// What the LLM held without experts.
pub model_mib: i64,
/// The long-prompt staging the expert budget holds first (engine log, "x.xx GiB staging").
pub staging_gib: f64,
}
/// The expert budget (GiB, down to 0.1) that leaves `room_mib` free besides what the card already
/// holds (`used_mib`), the LLM without experts, the STT's share and the engine's margin. The budget
/// first holds the staging, which the start without experts already has (M0 `expert_budget_gib`).
pub fn expert_budget_gib(total_mib: i64, used_mib: i64, p: &Probe, room_mib: i64) -> f64 {
let spare = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB - room_mib;
if spare <= 0 {
return 0.0;
}
((p.staging_gib * 1024.0 + spare as f64) / 1024.0 * 10.0).floor() / 10.0
}
#[derive(Debug, Clone, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct GamePlacement {
pub expert_gib: f64,
/// What a game really gets when the reserve does not fit beside the smallest placement.
pub game_room_mib: i64,
/// Not even the smallest placement leaves a game any room: the LLM pauses in a game.
pub paused: bool,
}
/// The normal placement: experts get what is left after the room for other programs opened later.
pub fn normal(total_mib: i64, used_mib: i64, p: &Probe, other_room_mib: i64) -> f64 {
expert_budget_gib(total_mib, used_mib, p, other_room_mib)
}
/// The game placement: experts get what is left after the game's reserve, but never less than the
/// smallest placement; then the game gets what that leaves (shown on the dashboard), and with no
/// room at all the LLM pauses. `p` is the probe at `GAME_CONTEXT`.
pub fn game(total_mib: i64, used_mib: i64, p: &Probe, reserve_mib: i64) -> GamePlacement {
let expert = expert_budget_gib(total_mib, used_mib, p, reserve_mib);
if expert >= MIN_GAME_EXPERT_GIB {
return GamePlacement { expert_gib: expert, game_room_mib: reserve_mib, paused: false };
}
let extra = ((MIN_GAME_EXPERT_GIB - p.staging_gib) * 1024.0).max(0.0) as i64;
let room = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB - extra;
GamePlacement { expert_gib: MIN_GAME_EXPERT_GIB, game_room_mib: room.max(0), paused: room <= 0 }
}
/// The staging the engine log reports ("(1.10 GiB staging)"), else the usual 1.1 GiB.
pub fn staging_gib(lines: &str) -> f64 {
lines.lines().rev().find_map(|l| {
let i = l.find(" GiB staging")?;
l[..i].rsplit(|c: char| !(c.is_ascii_digit() || c == '.')).next()?.parse().ok()
}).unwrap_or(1.1)
}
#[cfg(test)]
mod tests {
use super::*;
/// The game PC in M0 `gamemode_tune` (RTX 5070 Ti 16 GB): 2760 MiB held before the LLM there
/// included the STT model (1300 MiB), which the app starts after the LLM.
const TOTAL: i64 = 16303;
const OTHERS: i64 = 2760 - 1300;
const PROBE32: Probe = Probe { context: 32768, total_mib: TOTAL, model_mib: 9874, staging_gib: 1.1 };
#[test]
fn the_normal_placement_is_the_one_m0_measured_with_room_for_the_stt() {
// M0 got 1.5 GiB (the STT model counted as already held); the app counts it in the STT share.
assert_eq!(normal(TOTAL, OTHERS, &PROBE32, 1024), 1.5);
// More room for other programs, fewer experts; no room at all, none.
assert!(normal(TOTAL, OTHERS, &PROBE32, 2048) < 1.5);
assert_eq!(normal(TOTAL, OTHERS + 8000, &PROBE32, 1024), 0.0);
}
#[test]
fn a_game_gets_its_reserve_or_what_the_smallest_placement_leaves_or_the_llm_pauses() {
// 16K saves about 0.7 GB of conversation memory (M0).
let probe16 = Probe { context: GAME_CONTEXT, model_mib: 9874 - 750, ..PROBE32 };
// 8 GB does not fit on 16 GB: the smallest placement, and the game gets what is left.
let g = game(TOTAL, OTHERS, &probe16, 8192);
assert_eq!((g.expert_gib, g.paused), (MIN_GAME_EXPERT_GIB, false));
let expected = TOTAL - OTHERS - probe16.model_mib - STT_MIB - MARGIN_MIB - ((2.0 - 1.1) * 1024.0) as i64;
assert_eq!(g.game_room_mib, expected);
assert!(g.game_room_mib > 0 && g.game_room_mib < 8192);
// A bigger card keeps the whole reserve, with more experts than the minimum.
let big = game(24 * 1024, OTHERS, &Probe { total_mib: 24 * 1024, ..probe16 }, 8192);
assert_eq!(big.game_room_mib, 8192);
assert!(big.expert_gib > MIN_GAME_EXPERT_GIB && !big.paused);
// Other programs already hold the card: no room for a game, the LLM pauses in one.
let full = game(TOTAL, OTHERS + 3000, &probe16, 8192);
assert!(full.paused);
assert_eq!(full.game_room_mib, 0);
}
#[test]
fn the_staging_comes_from_the_engine_log() {
assert_eq!(staging_gib("strata-q36: hybrid experts: 5 of 256 per layer in VRAM (0.37 GiB), bigger prompt chunks streamed (1.10 GiB staging); 7.1 s\n"), 1.1);
assert_eq!(staging_gib("x\nstrata-q36: budget 2.00 GiB (1.25 GiB staging)"), 1.25);
assert_eq!(staging_gib("no such line"), 1.1);
}
}

View File

@@ -102,11 +102,19 @@ pub struct FeatureSettings {
/// Game detection (admin.spec.md, "Game detection"): on by default; the person's own games and
/// programs that are never games, as program file names (".exe" may be left out).
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct GameSettings {
pub detect: bool,
pub games: Vec<String>,
pub not_games: Vec<String>,
/// Graphics memory the normal placement leaves for programs opened later (GB, default 1).
pub other_room_gb: f64,
}
impl GameSettings {
pub fn other_room_mib(&self) -> i64 {
(self.other_room_gb * 1024.0).round() as i64
}
}
/// Background collection (admin.spec.md, "Background collection"); no targets by default.
@@ -202,7 +210,7 @@ impl Default for Settings {
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true, browser_share: false },
mcp_servers: vec![],
collect: CollectSettings { keep_days: 30, targets: vec![] },
game: GameSettings { detect: true, games: vec![], not_games: vec![] },
game: GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 },
}
}
}
@@ -563,8 +571,15 @@ fn parse_game(raw: &Map<String, Value>, d: &GameSettings) -> Result<GameSettings
}
Ok(v)
};
let other_room_gb = match g.and_then(|m| m.get("otherRoomGb")) {
None => d.other_room_gb,
Some(v) => match v.as_f64() {
Some(n) if (0.0..=8.0).contains(&n) => n,
_ => return err("game.otherRoomGb must be a number 0-8 (GB)"),
},
};
Ok(GameSettings { detect: get_bool(g, "detect", "game", d.detect)?, games: list("games", &d.games)?,
not_games: list("notGames", &d.not_games)? })
not_games: list("notGames", &d.not_games)?, other_room_gb })
}
fn parse_collect(raw: &Map<String, Value>, d: &CollectSettings) -> Result<CollectSettings, SettingsError> {
@@ -836,10 +851,11 @@ mod tests {
#[test]
fn game_detection_settings_are_read_and_checked() {
assert_eq!(parse_settings(&json!({})).unwrap().game, GameSettings { detect: true, games: vec![], not_games: vec![] });
let s = parse_settings(&json!({"game": {"detect": false, "games": [" MyGame.exe ", ""], "notGames": ["obs64"]}})).unwrap();
assert_eq!(s.game, GameSettings { detect: false, games: vec!["MyGame.exe".into()], not_games: vec!["obs64".into()] });
for bad in [json!({"game": {"games": ["C:\\x\\a.exe"]}}), json!({"game": {"notGames": "a"}}), json!({"game": {"detect": 1}})] {
assert_eq!(parse_settings(&json!({})).unwrap().game, GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 });
let s = parse_settings(&json!({"game": {"detect": false, "games": [" MyGame.exe ", ""], "notGames": ["obs64"], "otherRoomGb": 2.5}})).unwrap();
assert_eq!(s.game, GameSettings { detect: false, games: vec!["MyGame.exe".into()], not_games: vec!["obs64".into()], other_room_gb: 2.5 });
assert_eq!(s.game.other_room_mib(), 2560);
for bad in [json!({"game": {"games": ["C:\\x\\a.exe"]}}), json!({"game": {"notGames": "a"}}), json!({"game": {"detect": 1}}), json!({"game": {"otherRoomGb": 9}})] {
assert!(parse_settings(&bad).unwrap_err().0.contains("game."), "{bad}");
}
}

View File

@@ -133,6 +133,7 @@ pub fn fields() -> Vec<Field> {
// Game detection (admin.spec.md, "Game detection").
f("game.detect", Game, Bool, json!(true), "게임 감지"),
f("game.games", Game, TextList { max_items: 100, nullable: false }, json!([]), "게임으로 볼 프로그램(파일 이름, 예: MyGame.exe)"),
f("game.otherRoomGb", Game, Number { min: 0.0, max: 8.0 }, json!(1.0), "평소 다른 프로그램용으로 남길 VRAM(GB)"),
f("game.notGames", Game, TextList { max_items: 100, nullable: false }, json!([]), "게임이 아닌 프로그램(파일 이름)"),
]
}

View File

@@ -49,7 +49,7 @@ pub fn parallel(settings: &Settings) -> u32 {
/// `strata launch` for the services: the M0 kit's normal placement, models in `models/llm`, this
/// computer only. No pictures yet (nothing in v2 sends one; reading them takes graphics memory).
/// `--restart` replaces a server started with other choices; the same choices are reused.
pub fn launch_args(layout: &Layout, settings: &Settings) -> Vec<String> {
pub fn launch_args(layout: &Layout, settings: &Settings, sets: &[String]) -> Vec<String> {
let models = layout.data.join("models").join("llm").to_string_lossy().into_owned();
let mut a: Vec<String> = ["launch", "--family", FAMILY, "--model", MODEL, "--data-dir"].map(String::from).into();
a.push(models);
@@ -60,9 +60,59 @@ pub fn launch_args(layout: &Layout, settings: &Settings) -> Vec<String> {
a.push(v);
}
a.extend(["--no-api-key", "--restart", "--yes", "--no-tui"].map(String::from));
for s in sets {
a.push("--set".into());
a.push(s.clone());
}
a
}
/// The engine settings for an expert budget (`placement.rs`) and the M0 kit's prompt chunk.
fn placement_sets(expert_gib: f64) -> Vec<String> {
vec![format!("expert-vram={expert_gib}"), "prefill=2048".into()]
}
/// Graphics memory `(used, total)` in MiB of the first NVIDIA card, from `nvidia-smi`; None without one.
pub type GpuReader = Arc<dyn Fn() -> Option<(i64, i64)> + Send + Sync>;
fn nvidia_memory() -> Option<(i64, i64)> {
let mut cmd = std::process::Command::new("nvidia-smi");
cmd.args(["--query-gpu=memory.used,memory.total", "--format=csv,noheader,nounits"]);
#[cfg(windows)]
{
use std::os::windows::process::CommandExt;
cmd.creation_flags(0x0800_0000);
}
let out = cmd.output().ok()?;
let line = String::from_utf8_lossy(&out.stdout).lines().next()?.to_owned();
let (used, total) = line.split_once(',')?;
Some((used.trim().parse().ok()?, total.trim().parse().ok()?))
}
/// The newest engine log under Strata's folder, its end only (the placement lines of the last start).
fn engine_log(home: &Path) -> String {
fn walk(dir: &Path, depth: u32, best: &mut Option<(std::time::SystemTime, std::path::PathBuf)>) {
let Ok(entries) = std::fs::read_dir(dir) else { return };
for e in entries.flatten() {
let p = e.path();
if p.is_dir() && depth > 0 {
walk(&p, depth - 1, best);
} else if p.to_string_lossy().ends_with(".engine.log") {
if let Ok(t) = e.metadata().and_then(|m| m.modified()) {
if best.as_ref().is_none_or(|(b, _)| t > *b) {
*best = Some((t, p));
}
}
}
}
}
let mut best = None;
walk(home, 5, &mut best);
let Some((_, p)) = best else { return String::new() };
let bytes = std::fs::read(p).unwrap_or_default();
String::from_utf8_lossy(&bytes[bytes.len().saturating_sub(400_000)..]).into_owned()
}
/// How far a line says Strata got: its last `NN%` or `NN.N%`, else `X of Y GB` (or MB), which is
/// how Strata 0.1.59 reports the model download.
pub fn percent(line: &str) -> Option<f64> {
@@ -117,6 +167,18 @@ pub async fn wait_ready(base: &str, timeout: Duration, cancel: &AtomicBool) -> R
}
}
/// One short answer, so the engine has made its buffers before its memory is read (M0 kit).
async fn warm_up(base: &str) {
let Ok(client) = reqwest::Client::builder().timeout(Duration::from_secs(60)).build() else { return };
let model = match client.get(format!("{base}/models")).send().await {
Ok(r) => r.text().await.ok().and_then(|t| serde_json::from_str::<Value>(&t).ok())
.and_then(|v| v["data"][0]["id"].as_str().map(str::to_owned)),
Err(_) => None,
};
let body = json!({"model": model, "messages": [{"role": "user", "content": "안녕"}], "max_tokens": 8});
let _ = client.post(format!("{base}/chat/completions")).header("content-type", "application/json").body(body.to_string()).send().await;
}
/// The brain's settings (`config/brain.json`, the worker's `JARVIS_CONFIG_PATH`): the LLM, the
/// memory database inside the data folder, the tools of the features that are off and the enabled
/// MCP servers (without their environment values, `features.rs`). The app owns this file and writes
@@ -149,6 +211,8 @@ pub struct Llm {
/// The port the managed server was started on (to confirm it is gone after a stop).
port: Mutex<u16>,
stop_timeout: Duration,
/// The graphics card's memory (none: Strata places the model itself).
gpu: Option<GpuReader>,
}
/// How long the model may take to answer after `strata launch` returned (it loads in about 15 s
@@ -162,13 +226,68 @@ impl Llm {
// Only the data folder matters here (the Python sources are the install's business).
let layout = Layout { data: data.to_path_buf(), src: std::path::PathBuf::new() };
let exe = layout.strata_exe();
Self::with_exe(layout, exe)
Self::with_exe(layout, exe).with_gpu(Arc::new(nvidia_memory))
}
pub fn with_exe(layout: Layout, exe: std::path::PathBuf) -> Self {
Self { layout, exe, status: Mutex::new(LlmStatus { state: LlmState::Off, managed: false, percent: None, note: None }),
active: Mutex::new(None), cancel: Arc::new(AtomicBool::new(false)), port: Mutex::new(0),
stop_timeout: Duration::from_secs(120) }
stop_timeout: Duration::from_secs(120), gpu: None }
}
/// Place the model by the card's memory (`placement.rs`), read with `gpu`.
pub fn with_gpu(mut self, gpu: GpuReader) -> Self {
self.gpu = Some(gpu);
self
}
async fn read_gpu(&self) -> Option<(i64, i64)> {
let gpu = self.gpu.clone()?;
tokio::task::spawn_blocking(move || gpu()).await.ok().flatten()
}
fn probe_file(&self) -> std::path::PathBuf {
self.layout.data.join("config").join("llm-probe.json")
}
/// The expert budget's engine settings (none without a card): the normal placement from what
/// the card holds now and this PC's measurement without experts at this context, which the
/// first start makes once (a start without experts, then the real start replaces it).
async fn placement(&self, settings: &Settings, ctx: &Ctx, base: &str) -> Result<Vec<String>, String> {
let Some((used, total)) = self.read_gpu().await else { return Ok(vec![]) };
let context = settings.llm.context;
let file = self.probe_file();
let mut all: serde_json::Map<String, Value> = std::fs::read_to_string(&file).ok()
.and_then(|t| serde_json::from_str(&t).ok()).unwrap_or_default();
let known = all.get(&context.to_string()).and_then(|v| serde_json::from_value::<crate::placement::Probe>(v.clone()).ok())
.filter(|p| p.total_mib == total);
let probe = match known {
Some(p) => p,
None => {
crate::info_log!("llm", "measuring the model without experts once (context {context})");
self.launch(settings, ctx, base, &placement_sets(0.0)).await?;
warm_up(base).await;
let after = self.read_gpu().await.map(|(u, _)| u).unwrap_or(used);
let p = crate::placement::Probe { context, total_mib: total, model_mib: (after - used).max(0),
staging_gib: crate::placement::staging_gib(&engine_log(&self.layout.strata_home())) };
all.insert(context.to_string(), json!(p));
crate::datadir::write_atomic(&file, Value::Object(all).to_string().as_bytes())
.map_err(|e| format!("배치 측정값을 저장하지 못했습니다: {e}"))?;
p
}
};
let expert = crate::placement::normal(total, used, &probe, settings.game.other_room_mib());
crate::info_log!("llm", "placement: experts {expert} GiB (card {total} MiB, held by others {used} MiB, model without experts {} MiB)",
probe.model_mib);
Ok(placement_sets(expert))
}
/// `strata launch` with `sets`, then wait until the model answers.
async fn launch(&self, settings: &Settings, ctx: &Ctx, base: &str, sets: &[String]) -> Result<String, String> {
let args = launch_args(&self.layout, settings, sets);
let refs: Vec<&str> = args.iter().map(String::as_str).collect();
ctx.run_in(&self.exe, &refs, Some(&self.layout.strata_home())).await?;
wait_ready(base, READY_TIMEOUT, &self.cancel).await
}
/// How long `strata launch --stop` may take before it is ended and the stop reported as failed.
@@ -229,13 +348,9 @@ impl Llm {
let ctx = Arc::new(Ctx::standalone(self.layout.clone(), "llm", self.cancel.clone()));
*self.active.lock().unwrap() = Some(ctx.clone());
*self.port.lock().unwrap() = settings.llm.port;
let args = launch_args(&self.layout, settings);
let refs: Vec<&str> = args.iter().map(String::as_str).collect();
let home = self.layout.strata_home();
let ran = ctx.run_in(&self.exe, &refs, Some(&home)).await;
let base = format!("http://127.0.0.1:{}/v1", settings.llm.port);
let ready = match ran {
Ok(()) => wait_ready(&base, READY_TIMEOUT, &self.cancel).await,
let ready = match self.placement(settings, &ctx, &base).await {
Ok(sets) => self.launch(settings, &ctx, &base, &sets).await,
Err(e) => Err(e),
};
*self.active.lock().unwrap() = None;
@@ -350,7 +465,7 @@ mod tests {
fn the_launch_keeps_the_model_in_the_data_folder_and_follows_the_settings() {
let l = layout("llm1");
let s = parse_settings(&json!({"llm": {"context": 16384, "port": 8099}})).unwrap();
let a = launch_args(&l, &s);
let a = launch_args(&l, &s, &[]);
let after = |k: &str| a[a.iter().position(|x| x == k).unwrap() + 1].clone();
assert!(Path::new(&after("--data-dir")).starts_with(&l.data));
assert_eq!((after("--context"), after("--port"), after("--parallel")), ("16384".into(), "8099".into(), "1".into()));
@@ -526,6 +641,46 @@ mod tests {
assert_eq!(llm.status().state, LlmState::Off);
}
#[cfg(unix)]
#[tokio::test]
async fn with_a_graphics_card_the_first_start_measures_the_llm_without_experts_then_places_them() {
let l = layout("llm-place");
let port = models_server_until("qwen-test", Duration::from_millis(0), Some(l.data.join("gone"))).await;
// The engine log, as Strata 0.1.59's qwen36 engine writes it.
let log = l.strata_home().join("x.engine.log");
let exe = fake_strata(&l, &format!("rm -f '{}'; echo 'strata-q36: hybrid experts: 0 of 256 per layer in VRAM, bigger prompt chunks streamed (1.10 GiB staging); 7.1 s' >> '{}'; echo 'Strata is serving.'",
l.data.join("gone").display(), log.display()));
// The game PC in M0 (16303 MiB card, 1460 MiB held by other programs); the LLM without
// experts held 9874 MiB.
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
let r2 = reads.clone();
let llm = Llm::with_exe(l.clone(), exe).with_gpu(Arc::new(move || {
let n = r2.fetch_add(1, Ordering::SeqCst);
Some((if n == 1 { 1460 + 9874 } else { 1460 }, 16303))
}));
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
let calls = std::fs::read_to_string(l.data.join("calls")).unwrap();
let launches: Vec<&str> = calls.lines().filter(|c| c.starts_with("launch --family")).collect();
assert_eq!(launches.len(), 2, "{calls}");
assert!(launches[0].ends_with("--set expert-vram=0 --set prefill=2048"), "{}", launches[0]);
// The M0 kit's normal placement on that PC: 1.5 GiB of experts.
assert!(launches[1].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[1]);
let probe: Value = serde_json::from_str(&std::fs::read_to_string(l.data.join("config").join("llm-probe.json")).unwrap()).unwrap();
assert_eq!(probe["32768"], json!({"context": 32768, "totalMib": 16303, "modelMib": 9874, "stagingGib": 1.1}));
llm.stop().await.unwrap();
// The next start reuses the measurement: one launch, placed at once (a new server, as the
// fake one does not come back after a stop).
std::fs::remove_file(l.data.join("calls")).unwrap();
let port2 = models_server_until("qwen-test", Duration::from_millis(0), None).await;
let s2 = parse_settings(&json!({"llm": {"port": port2}})).unwrap();
tokio::time::timeout(Duration::from_secs(20), llm.start(&s2)).await.expect("starts").unwrap();
let calls = std::fs::read_to_string(l.data.join("calls")).unwrap();
let launches: Vec<&str> = calls.lines().filter(|c| c.starts_with("launch --family")).collect();
assert_eq!(launches.len(), 1, "{calls}");
assert!(launches[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[0]);
}
#[cfg(unix)]
#[tokio::test]
async fn a_cancel_during_the_download_stops_it_and_clears_the_server() {

View File

@@ -26,7 +26,7 @@ fn a_launcher_game_turns_game_mode_on_until_it_ends() {
eprintln!("skipped: set JARVIS_TEST_GAME_EXE");
return;
};
let w = GameWatch::start(GameSettings { detect: true, games: vec![], not_games: vec![] });
let w = GameWatch::start(GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 });
assert_eq!(w.view(), serde_json::Value::Null, "no game before it starts");
let mut game = std::process::Command::new(&exe).spawn().expect("the game starts");
let t0 = Instant::now();

File diff suppressed because one or more lines are too long