feat(app): place the LLM's experts so the speech-to-text worker keeps its room
- with an NVIDIA card the app sets --expert-vram itself by the M0 kit's formula: what the card holds before the start, the model without experts, the STT worker's share (2740 MiB), the engine's margin (768 MiB) and room for programs opened later (game.otherRoomGb, default 1 GB) are left free; Strata's own placement filled the card and left the STT no room - the model without experts is measured once per context and card (a start with expert-vram=0, one short answer, the card's memory and the engine log's staging), kept in config/llm-probe.json; the real start replaces that server - placement.rs also works out the game placement (the reserve, else the 2 GiB minimum and the room a game really gets, else pause) for the switching that comes next - with the game PC's M0 numbers the normal placement comes out at 1.5 GiB, as M0 measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -335,7 +335,8 @@ elsewhere it never sees a game). It never looks at the game's screen.
|
||||
|
||||
- `game` (new section): `detect` (게임 감지, on by default), `games` (programs that are games, as
|
||||
file names, ".exe" may be left out) and `notGames` (programs that are never games); each list at
|
||||
most 100 names, no folders. Read when the services start.
|
||||
most 100 names, no folders; `otherRoomGb` (0-8, default 1), the graphics memory the normal
|
||||
placement leaves for programs opened later (above, "LLM"). Read when the services start.
|
||||
- The foreground program is a game, in this order: in `notGames`: no; in `games`: yes
|
||||
(`registered`); a browser (also the bot's Chrome), video player, the desktop, Discord, a
|
||||
launcher's own window or this app: no; under a launcher's game folder (the folder after
|
||||
@@ -377,6 +378,18 @@ start it first, so nothing answers before the brain can, and stop it last.
|
||||
accepts the start (`Host::prepare_start`), never by the start itself, so a stop between accepting
|
||||
and running the start cancels it before Strata runs anything. Not installed: the start fails and says to
|
||||
run the install.
|
||||
- **Placement** (`app/src/placement.rs`, `docs/v2/pc-features.md` "PC마다 값을 계산"): with an
|
||||
NVIDIA card (`nvidia-smi`) the app sets the routed experts' graphics memory itself, `--set
|
||||
expert-vram=<GiB> --set prefill=2048`, as the M0 kit did; Strata's own placement fills the card
|
||||
and leaves the speech-to-text worker no room, and its qwen36 engine takes no reserve. Experts get
|
||||
what is left of the card after what it holds before the start (other programs), the model without
|
||||
experts, the STT worker's share (2740 MiB: its model and ten speakers at once, M0 `stt`), the
|
||||
engine's margin (768 MiB) and the room for programs opened later (`game.otherRoomGb`, default
|
||||
1 GB); the budget first holds the long-prompt staging, rounded down to 0.1 GiB, 0 when nothing is
|
||||
left. What the model holds without experts is measured once per context and card (the first-run
|
||||
self-check: a start with `expert-vram=0`, one short answer, then the card's memory again and the
|
||||
staging from the engine log), kept in `config/llm-probe.json`; the real start then replaces that
|
||||
server. Without a card nothing is set. The `llm` log tab gets the placement and what it came from.
|
||||
- **A server the app does not start**: with `llm.url` (an OpenAI-compatible `/v1` address, in the
|
||||
settings file only) the app starts nothing; that server must list a model within 30 seconds.
|
||||
- **Settings**: `llm.context` (default 32768 tokens per request, 4096-262144) and `llm.port`
|
||||
|
||||
@@ -479,7 +479,7 @@ mod tests {
|
||||
use super::*;
|
||||
|
||||
fn settings() -> GameSettings {
|
||||
GameSettings { detect: true, games: vec![], not_games: vec![] }
|
||||
GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 }
|
||||
}
|
||||
|
||||
fn fg(path: &str, covers: bool, no_frame: bool, gpu: f64) -> Foreground {
|
||||
|
||||
@@ -21,6 +21,7 @@ pub mod frames;
|
||||
pub mod install;
|
||||
pub mod logs;
|
||||
pub mod playback;
|
||||
pub mod placement;
|
||||
pub mod policy;
|
||||
pub mod receive;
|
||||
pub mod secrets;
|
||||
|
||||
126
app/src/placement.rs
Normal file
126
app/src/placement.rs
Normal file
@@ -0,0 +1,126 @@
|
||||
//! Where the LLM's routed experts go (`docs/v2/pc-features.md`, "PC마다 값을 계산"; admin.spec.md,
|
||||
//! "LLM placement"): the one model is placed by the M0 kit's formula, so the speech-to-text worker
|
||||
//! and other programs keep their room on the graphics card. Strata's automatic placement fills the
|
||||
//! card with experts and left the STT no room (M0 `gamemode_stt`), and its qwen36 engine takes no
|
||||
//! VRAM reserve, so the app sets `--expert-vram` itself.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// What Strata's qwen36 engine keeps free itself when it places experts (M0 kit).
|
||||
pub const MARGIN_MIB: i64 = 768;
|
||||
/// The STT worker's share: its model (1300 MiB) and what ten speakers at once add (1440 MiB),
|
||||
/// measured on the game PC (M0 `stt`). The services start the LLM before the worker, so both count.
|
||||
pub const STT_MIB: i64 = 1300 + 1440;
|
||||
/// The smallest expert budget in a game: it holds the long-prompt staging (about 1.1 GiB) first,
|
||||
/// so 2 GiB is the least that loads experts, and it was 1.5-2.1 times faster than none in a real
|
||||
/// game (user decision, 2026-11-10).
|
||||
pub const MIN_GAME_EXPERT_GIB: f64 = 2.0;
|
||||
/// The context in a game (16K keeps room for the history, tools and results; 8K saved only 0.4 GB).
|
||||
pub const GAME_CONTEXT: u32 = 16384;
|
||||
|
||||
/// What one start without experts at `context` showed on this PC (the first-run self-check).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct Probe {
|
||||
pub context: u32,
|
||||
/// The card's size when it was measured (another card means measuring again).
|
||||
pub total_mib: i64,
|
||||
/// What the LLM held without experts.
|
||||
pub model_mib: i64,
|
||||
/// The long-prompt staging the expert budget holds first (engine log, "x.xx GiB staging").
|
||||
pub staging_gib: f64,
|
||||
}
|
||||
|
||||
/// The expert budget (GiB, down to 0.1) that leaves `room_mib` free besides what the card already
|
||||
/// holds (`used_mib`), the LLM without experts, the STT's share and the engine's margin. The budget
|
||||
/// first holds the staging, which the start without experts already has (M0 `expert_budget_gib`).
|
||||
pub fn expert_budget_gib(total_mib: i64, used_mib: i64, p: &Probe, room_mib: i64) -> f64 {
|
||||
let spare = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB - room_mib;
|
||||
if spare <= 0 {
|
||||
return 0.0;
|
||||
}
|
||||
((p.staging_gib * 1024.0 + spare as f64) / 1024.0 * 10.0).floor() / 10.0
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct GamePlacement {
|
||||
pub expert_gib: f64,
|
||||
/// What a game really gets when the reserve does not fit beside the smallest placement.
|
||||
pub game_room_mib: i64,
|
||||
/// Not even the smallest placement leaves a game any room: the LLM pauses in a game.
|
||||
pub paused: bool,
|
||||
}
|
||||
|
||||
/// The normal placement: experts get what is left after the room for other programs opened later.
|
||||
pub fn normal(total_mib: i64, used_mib: i64, p: &Probe, other_room_mib: i64) -> f64 {
|
||||
expert_budget_gib(total_mib, used_mib, p, other_room_mib)
|
||||
}
|
||||
|
||||
/// The game placement: experts get what is left after the game's reserve, but never less than the
|
||||
/// smallest placement; then the game gets what that leaves (shown on the dashboard), and with no
|
||||
/// room at all the LLM pauses. `p` is the probe at `GAME_CONTEXT`.
|
||||
pub fn game(total_mib: i64, used_mib: i64, p: &Probe, reserve_mib: i64) -> GamePlacement {
|
||||
let expert = expert_budget_gib(total_mib, used_mib, p, reserve_mib);
|
||||
if expert >= MIN_GAME_EXPERT_GIB {
|
||||
return GamePlacement { expert_gib: expert, game_room_mib: reserve_mib, paused: false };
|
||||
}
|
||||
let extra = ((MIN_GAME_EXPERT_GIB - p.staging_gib) * 1024.0).max(0.0) as i64;
|
||||
let room = total_mib - used_mib - p.model_mib - STT_MIB - MARGIN_MIB - extra;
|
||||
GamePlacement { expert_gib: MIN_GAME_EXPERT_GIB, game_room_mib: room.max(0), paused: room <= 0 }
|
||||
}
|
||||
|
||||
/// The staging the engine log reports ("(1.10 GiB staging)"), else the usual 1.1 GiB.
|
||||
pub fn staging_gib(lines: &str) -> f64 {
|
||||
lines.lines().rev().find_map(|l| {
|
||||
let i = l.find(" GiB staging")?;
|
||||
l[..i].rsplit(|c: char| !(c.is_ascii_digit() || c == '.')).next()?.parse().ok()
|
||||
}).unwrap_or(1.1)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The game PC in M0 `gamemode_tune` (RTX 5070 Ti 16 GB): 2760 MiB held before the LLM there
|
||||
/// included the STT model (1300 MiB), which the app starts after the LLM.
|
||||
const TOTAL: i64 = 16303;
|
||||
const OTHERS: i64 = 2760 - 1300;
|
||||
const PROBE32: Probe = Probe { context: 32768, total_mib: TOTAL, model_mib: 9874, staging_gib: 1.1 };
|
||||
|
||||
#[test]
|
||||
fn the_normal_placement_is_the_one_m0_measured_with_room_for_the_stt() {
|
||||
// M0 got 1.5 GiB (the STT model counted as already held); the app counts it in the STT share.
|
||||
assert_eq!(normal(TOTAL, OTHERS, &PROBE32, 1024), 1.5);
|
||||
// More room for other programs, fewer experts; no room at all, none.
|
||||
assert!(normal(TOTAL, OTHERS, &PROBE32, 2048) < 1.5);
|
||||
assert_eq!(normal(TOTAL, OTHERS + 8000, &PROBE32, 1024), 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_game_gets_its_reserve_or_what_the_smallest_placement_leaves_or_the_llm_pauses() {
|
||||
// 16K saves about 0.7 GB of conversation memory (M0).
|
||||
let probe16 = Probe { context: GAME_CONTEXT, model_mib: 9874 - 750, ..PROBE32 };
|
||||
// 8 GB does not fit on 16 GB: the smallest placement, and the game gets what is left.
|
||||
let g = game(TOTAL, OTHERS, &probe16, 8192);
|
||||
assert_eq!((g.expert_gib, g.paused), (MIN_GAME_EXPERT_GIB, false));
|
||||
let expected = TOTAL - OTHERS - probe16.model_mib - STT_MIB - MARGIN_MIB - ((2.0 - 1.1) * 1024.0) as i64;
|
||||
assert_eq!(g.game_room_mib, expected);
|
||||
assert!(g.game_room_mib > 0 && g.game_room_mib < 8192);
|
||||
// A bigger card keeps the whole reserve, with more experts than the minimum.
|
||||
let big = game(24 * 1024, OTHERS, &Probe { total_mib: 24 * 1024, ..probe16 }, 8192);
|
||||
assert_eq!(big.game_room_mib, 8192);
|
||||
assert!(big.expert_gib > MIN_GAME_EXPERT_GIB && !big.paused);
|
||||
// Other programs already hold the card: no room for a game, the LLM pauses in one.
|
||||
let full = game(TOTAL, OTHERS + 3000, &probe16, 8192);
|
||||
assert!(full.paused);
|
||||
assert_eq!(full.game_room_mib, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_staging_comes_from_the_engine_log() {
|
||||
assert_eq!(staging_gib("strata-q36: hybrid experts: 5 of 256 per layer in VRAM (0.37 GiB), bigger prompt chunks streamed (1.10 GiB staging); 7.1 s\n"), 1.1);
|
||||
assert_eq!(staging_gib("x\nstrata-q36: budget 2.00 GiB (1.25 GiB staging)"), 1.25);
|
||||
assert_eq!(staging_gib("no such line"), 1.1);
|
||||
}
|
||||
}
|
||||
@@ -102,11 +102,19 @@ pub struct FeatureSettings {
|
||||
|
||||
/// Game detection (admin.spec.md, "Game detection"): on by default; the person's own games and
|
||||
/// programs that are never games, as program file names (".exe" may be left out).
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct GameSettings {
|
||||
pub detect: bool,
|
||||
pub games: Vec<String>,
|
||||
pub not_games: Vec<String>,
|
||||
/// Graphics memory the normal placement leaves for programs opened later (GB, default 1).
|
||||
pub other_room_gb: f64,
|
||||
}
|
||||
|
||||
impl GameSettings {
|
||||
pub fn other_room_mib(&self) -> i64 {
|
||||
(self.other_room_gb * 1024.0).round() as i64
|
||||
}
|
||||
}
|
||||
|
||||
/// Background collection (admin.spec.md, "Background collection"); no targets by default.
|
||||
@@ -202,7 +210,7 @@ impl Default for Settings {
|
||||
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true, browser_share: false },
|
||||
mcp_servers: vec![],
|
||||
collect: CollectSettings { keep_days: 30, targets: vec![] },
|
||||
game: GameSettings { detect: true, games: vec![], not_games: vec![] },
|
||||
game: GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 },
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -563,8 +571,15 @@ fn parse_game(raw: &Map<String, Value>, d: &GameSettings) -> Result<GameSettings
|
||||
}
|
||||
Ok(v)
|
||||
};
|
||||
let other_room_gb = match g.and_then(|m| m.get("otherRoomGb")) {
|
||||
None => d.other_room_gb,
|
||||
Some(v) => match v.as_f64() {
|
||||
Some(n) if (0.0..=8.0).contains(&n) => n,
|
||||
_ => return err("game.otherRoomGb must be a number 0-8 (GB)"),
|
||||
},
|
||||
};
|
||||
Ok(GameSettings { detect: get_bool(g, "detect", "game", d.detect)?, games: list("games", &d.games)?,
|
||||
not_games: list("notGames", &d.not_games)? })
|
||||
not_games: list("notGames", &d.not_games)?, other_room_gb })
|
||||
}
|
||||
|
||||
fn parse_collect(raw: &Map<String, Value>, d: &CollectSettings) -> Result<CollectSettings, SettingsError> {
|
||||
@@ -836,10 +851,11 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn game_detection_settings_are_read_and_checked() {
|
||||
assert_eq!(parse_settings(&json!({})).unwrap().game, GameSettings { detect: true, games: vec![], not_games: vec![] });
|
||||
let s = parse_settings(&json!({"game": {"detect": false, "games": [" MyGame.exe ", ""], "notGames": ["obs64"]}})).unwrap();
|
||||
assert_eq!(s.game, GameSettings { detect: false, games: vec!["MyGame.exe".into()], not_games: vec!["obs64".into()] });
|
||||
for bad in [json!({"game": {"games": ["C:\\x\\a.exe"]}}), json!({"game": {"notGames": "a"}}), json!({"game": {"detect": 1}})] {
|
||||
assert_eq!(parse_settings(&json!({})).unwrap().game, GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 });
|
||||
let s = parse_settings(&json!({"game": {"detect": false, "games": [" MyGame.exe ", ""], "notGames": ["obs64"], "otherRoomGb": 2.5}})).unwrap();
|
||||
assert_eq!(s.game, GameSettings { detect: false, games: vec!["MyGame.exe".into()], not_games: vec!["obs64".into()], other_room_gb: 2.5 });
|
||||
assert_eq!(s.game.other_room_mib(), 2560);
|
||||
for bad in [json!({"game": {"games": ["C:\\x\\a.exe"]}}), json!({"game": {"notGames": "a"}}), json!({"game": {"detect": 1}}), json!({"game": {"otherRoomGb": 9}})] {
|
||||
assert!(parse_settings(&bad).unwrap_err().0.contains("game."), "{bad}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -133,6 +133,7 @@ pub fn fields() -> Vec<Field> {
|
||||
// Game detection (admin.spec.md, "Game detection").
|
||||
f("game.detect", Game, Bool, json!(true), "게임 감지"),
|
||||
f("game.games", Game, TextList { max_items: 100, nullable: false }, json!([]), "게임으로 볼 프로그램(파일 이름, 예: MyGame.exe)"),
|
||||
f("game.otherRoomGb", Game, Number { min: 0.0, max: 8.0 }, json!(1.0), "평소 다른 프로그램용으로 남길 VRAM(GB)"),
|
||||
f("game.notGames", Game, TextList { max_items: 100, nullable: false }, json!([]), "게임이 아닌 프로그램(파일 이름)"),
|
||||
]
|
||||
}
|
||||
|
||||
@@ -49,7 +49,7 @@ pub fn parallel(settings: &Settings) -> u32 {
|
||||
/// `strata launch` for the services: the M0 kit's normal placement, models in `models/llm`, this
|
||||
/// computer only. No pictures yet (nothing in v2 sends one; reading them takes graphics memory).
|
||||
/// `--restart` replaces a server started with other choices; the same choices are reused.
|
||||
pub fn launch_args(layout: &Layout, settings: &Settings) -> Vec<String> {
|
||||
pub fn launch_args(layout: &Layout, settings: &Settings, sets: &[String]) -> Vec<String> {
|
||||
let models = layout.data.join("models").join("llm").to_string_lossy().into_owned();
|
||||
let mut a: Vec<String> = ["launch", "--family", FAMILY, "--model", MODEL, "--data-dir"].map(String::from).into();
|
||||
a.push(models);
|
||||
@@ -60,9 +60,59 @@ pub fn launch_args(layout: &Layout, settings: &Settings) -> Vec<String> {
|
||||
a.push(v);
|
||||
}
|
||||
a.extend(["--no-api-key", "--restart", "--yes", "--no-tui"].map(String::from));
|
||||
for s in sets {
|
||||
a.push("--set".into());
|
||||
a.push(s.clone());
|
||||
}
|
||||
a
|
||||
}
|
||||
|
||||
/// The engine settings for an expert budget (`placement.rs`) and the M0 kit's prompt chunk.
|
||||
fn placement_sets(expert_gib: f64) -> Vec<String> {
|
||||
vec![format!("expert-vram={expert_gib}"), "prefill=2048".into()]
|
||||
}
|
||||
|
||||
/// Graphics memory `(used, total)` in MiB of the first NVIDIA card, from `nvidia-smi`; None without one.
|
||||
pub type GpuReader = Arc<dyn Fn() -> Option<(i64, i64)> + Send + Sync>;
|
||||
|
||||
fn nvidia_memory() -> Option<(i64, i64)> {
|
||||
let mut cmd = std::process::Command::new("nvidia-smi");
|
||||
cmd.args(["--query-gpu=memory.used,memory.total", "--format=csv,noheader,nounits"]);
|
||||
#[cfg(windows)]
|
||||
{
|
||||
use std::os::windows::process::CommandExt;
|
||||
cmd.creation_flags(0x0800_0000);
|
||||
}
|
||||
let out = cmd.output().ok()?;
|
||||
let line = String::from_utf8_lossy(&out.stdout).lines().next()?.to_owned();
|
||||
let (used, total) = line.split_once(',')?;
|
||||
Some((used.trim().parse().ok()?, total.trim().parse().ok()?))
|
||||
}
|
||||
|
||||
/// The newest engine log under Strata's folder, its end only (the placement lines of the last start).
|
||||
fn engine_log(home: &Path) -> String {
|
||||
fn walk(dir: &Path, depth: u32, best: &mut Option<(std::time::SystemTime, std::path::PathBuf)>) {
|
||||
let Ok(entries) = std::fs::read_dir(dir) else { return };
|
||||
for e in entries.flatten() {
|
||||
let p = e.path();
|
||||
if p.is_dir() && depth > 0 {
|
||||
walk(&p, depth - 1, best);
|
||||
} else if p.to_string_lossy().ends_with(".engine.log") {
|
||||
if let Ok(t) = e.metadata().and_then(|m| m.modified()) {
|
||||
if best.as_ref().is_none_or(|(b, _)| t > *b) {
|
||||
*best = Some((t, p));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut best = None;
|
||||
walk(home, 5, &mut best);
|
||||
let Some((_, p)) = best else { return String::new() };
|
||||
let bytes = std::fs::read(p).unwrap_or_default();
|
||||
String::from_utf8_lossy(&bytes[bytes.len().saturating_sub(400_000)..]).into_owned()
|
||||
}
|
||||
|
||||
/// How far a line says Strata got: its last `NN%` or `NN.N%`, else `X of Y GB` (or MB), which is
|
||||
/// how Strata 0.1.59 reports the model download.
|
||||
pub fn percent(line: &str) -> Option<f64> {
|
||||
@@ -117,6 +167,18 @@ pub async fn wait_ready(base: &str, timeout: Duration, cancel: &AtomicBool) -> R
|
||||
}
|
||||
}
|
||||
|
||||
/// One short answer, so the engine has made its buffers before its memory is read (M0 kit).
|
||||
async fn warm_up(base: &str) {
|
||||
let Ok(client) = reqwest::Client::builder().timeout(Duration::from_secs(60)).build() else { return };
|
||||
let model = match client.get(format!("{base}/models")).send().await {
|
||||
Ok(r) => r.text().await.ok().and_then(|t| serde_json::from_str::<Value>(&t).ok())
|
||||
.and_then(|v| v["data"][0]["id"].as_str().map(str::to_owned)),
|
||||
Err(_) => None,
|
||||
};
|
||||
let body = json!({"model": model, "messages": [{"role": "user", "content": "안녕"}], "max_tokens": 8});
|
||||
let _ = client.post(format!("{base}/chat/completions")).header("content-type", "application/json").body(body.to_string()).send().await;
|
||||
}
|
||||
|
||||
/// The brain's settings (`config/brain.json`, the worker's `JARVIS_CONFIG_PATH`): the LLM, the
|
||||
/// memory database inside the data folder, the tools of the features that are off and the enabled
|
||||
/// MCP servers (without their environment values, `features.rs`). The app owns this file and writes
|
||||
@@ -149,6 +211,8 @@ pub struct Llm {
|
||||
/// The port the managed server was started on (to confirm it is gone after a stop).
|
||||
port: Mutex<u16>,
|
||||
stop_timeout: Duration,
|
||||
/// The graphics card's memory (none: Strata places the model itself).
|
||||
gpu: Option<GpuReader>,
|
||||
}
|
||||
|
||||
/// How long the model may take to answer after `strata launch` returned (it loads in about 15 s
|
||||
@@ -162,13 +226,68 @@ impl Llm {
|
||||
// Only the data folder matters here (the Python sources are the install's business).
|
||||
let layout = Layout { data: data.to_path_buf(), src: std::path::PathBuf::new() };
|
||||
let exe = layout.strata_exe();
|
||||
Self::with_exe(layout, exe)
|
||||
Self::with_exe(layout, exe).with_gpu(Arc::new(nvidia_memory))
|
||||
}
|
||||
|
||||
pub fn with_exe(layout: Layout, exe: std::path::PathBuf) -> Self {
|
||||
Self { layout, exe, status: Mutex::new(LlmStatus { state: LlmState::Off, managed: false, percent: None, note: None }),
|
||||
active: Mutex::new(None), cancel: Arc::new(AtomicBool::new(false)), port: Mutex::new(0),
|
||||
stop_timeout: Duration::from_secs(120) }
|
||||
stop_timeout: Duration::from_secs(120), gpu: None }
|
||||
}
|
||||
|
||||
/// Place the model by the card's memory (`placement.rs`), read with `gpu`.
|
||||
pub fn with_gpu(mut self, gpu: GpuReader) -> Self {
|
||||
self.gpu = Some(gpu);
|
||||
self
|
||||
}
|
||||
|
||||
async fn read_gpu(&self) -> Option<(i64, i64)> {
|
||||
let gpu = self.gpu.clone()?;
|
||||
tokio::task::spawn_blocking(move || gpu()).await.ok().flatten()
|
||||
}
|
||||
|
||||
fn probe_file(&self) -> std::path::PathBuf {
|
||||
self.layout.data.join("config").join("llm-probe.json")
|
||||
}
|
||||
|
||||
/// The expert budget's engine settings (none without a card): the normal placement from what
|
||||
/// the card holds now and this PC's measurement without experts at this context, which the
|
||||
/// first start makes once (a start without experts, then the real start replaces it).
|
||||
async fn placement(&self, settings: &Settings, ctx: &Ctx, base: &str) -> Result<Vec<String>, String> {
|
||||
let Some((used, total)) = self.read_gpu().await else { return Ok(vec![]) };
|
||||
let context = settings.llm.context;
|
||||
let file = self.probe_file();
|
||||
let mut all: serde_json::Map<String, Value> = std::fs::read_to_string(&file).ok()
|
||||
.and_then(|t| serde_json::from_str(&t).ok()).unwrap_or_default();
|
||||
let known = all.get(&context.to_string()).and_then(|v| serde_json::from_value::<crate::placement::Probe>(v.clone()).ok())
|
||||
.filter(|p| p.total_mib == total);
|
||||
let probe = match known {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
crate::info_log!("llm", "measuring the model without experts once (context {context})");
|
||||
self.launch(settings, ctx, base, &placement_sets(0.0)).await?;
|
||||
warm_up(base).await;
|
||||
let after = self.read_gpu().await.map(|(u, _)| u).unwrap_or(used);
|
||||
let p = crate::placement::Probe { context, total_mib: total, model_mib: (after - used).max(0),
|
||||
staging_gib: crate::placement::staging_gib(&engine_log(&self.layout.strata_home())) };
|
||||
all.insert(context.to_string(), json!(p));
|
||||
crate::datadir::write_atomic(&file, Value::Object(all).to_string().as_bytes())
|
||||
.map_err(|e| format!("배치 측정값을 저장하지 못했습니다: {e}"))?;
|
||||
p
|
||||
}
|
||||
};
|
||||
let expert = crate::placement::normal(total, used, &probe, settings.game.other_room_mib());
|
||||
crate::info_log!("llm", "placement: experts {expert} GiB (card {total} MiB, held by others {used} MiB, model without experts {} MiB)",
|
||||
probe.model_mib);
|
||||
Ok(placement_sets(expert))
|
||||
}
|
||||
|
||||
/// `strata launch` with `sets`, then wait until the model answers.
|
||||
async fn launch(&self, settings: &Settings, ctx: &Ctx, base: &str, sets: &[String]) -> Result<String, String> {
|
||||
let args = launch_args(&self.layout, settings, sets);
|
||||
let refs: Vec<&str> = args.iter().map(String::as_str).collect();
|
||||
ctx.run_in(&self.exe, &refs, Some(&self.layout.strata_home())).await?;
|
||||
wait_ready(base, READY_TIMEOUT, &self.cancel).await
|
||||
}
|
||||
|
||||
/// How long `strata launch --stop` may take before it is ended and the stop reported as failed.
|
||||
@@ -229,13 +348,9 @@ impl Llm {
|
||||
let ctx = Arc::new(Ctx::standalone(self.layout.clone(), "llm", self.cancel.clone()));
|
||||
*self.active.lock().unwrap() = Some(ctx.clone());
|
||||
*self.port.lock().unwrap() = settings.llm.port;
|
||||
let args = launch_args(&self.layout, settings);
|
||||
let refs: Vec<&str> = args.iter().map(String::as_str).collect();
|
||||
let home = self.layout.strata_home();
|
||||
let ran = ctx.run_in(&self.exe, &refs, Some(&home)).await;
|
||||
let base = format!("http://127.0.0.1:{}/v1", settings.llm.port);
|
||||
let ready = match ran {
|
||||
Ok(()) => wait_ready(&base, READY_TIMEOUT, &self.cancel).await,
|
||||
let ready = match self.placement(settings, &ctx, &base).await {
|
||||
Ok(sets) => self.launch(settings, &ctx, &base, &sets).await,
|
||||
Err(e) => Err(e),
|
||||
};
|
||||
*self.active.lock().unwrap() = None;
|
||||
@@ -350,7 +465,7 @@ mod tests {
|
||||
fn the_launch_keeps_the_model_in_the_data_folder_and_follows_the_settings() {
|
||||
let l = layout("llm1");
|
||||
let s = parse_settings(&json!({"llm": {"context": 16384, "port": 8099}})).unwrap();
|
||||
let a = launch_args(&l, &s);
|
||||
let a = launch_args(&l, &s, &[]);
|
||||
let after = |k: &str| a[a.iter().position(|x| x == k).unwrap() + 1].clone();
|
||||
assert!(Path::new(&after("--data-dir")).starts_with(&l.data));
|
||||
assert_eq!((after("--context"), after("--port"), after("--parallel")), ("16384".into(), "8099".into(), "1".into()));
|
||||
@@ -526,6 +641,46 @@ mod tests {
|
||||
assert_eq!(llm.status().state, LlmState::Off);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn with_a_graphics_card_the_first_start_measures_the_llm_without_experts_then_places_them() {
|
||||
let l = layout("llm-place");
|
||||
let port = models_server_until("qwen-test", Duration::from_millis(0), Some(l.data.join("gone"))).await;
|
||||
// The engine log, as Strata 0.1.59's qwen36 engine writes it.
|
||||
let log = l.strata_home().join("x.engine.log");
|
||||
let exe = fake_strata(&l, &format!("rm -f '{}'; echo 'strata-q36: hybrid experts: 0 of 256 per layer in VRAM, bigger prompt chunks streamed (1.10 GiB staging); 7.1 s' >> '{}'; echo 'Strata is serving.'",
|
||||
l.data.join("gone").display(), log.display()));
|
||||
// The game PC in M0 (16303 MiB card, 1460 MiB held by other programs); the LLM without
|
||||
// experts held 9874 MiB.
|
||||
let reads = Arc::new(std::sync::atomic::AtomicU32::new(0));
|
||||
let r2 = reads.clone();
|
||||
let llm = Llm::with_exe(l.clone(), exe).with_gpu(Arc::new(move || {
|
||||
let n = r2.fetch_add(1, Ordering::SeqCst);
|
||||
Some((if n == 1 { 1460 + 9874 } else { 1460 }, 16303))
|
||||
}));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(20), llm.start(&s)).await.expect("starts").unwrap();
|
||||
let calls = std::fs::read_to_string(l.data.join("calls")).unwrap();
|
||||
let launches: Vec<&str> = calls.lines().filter(|c| c.starts_with("launch --family")).collect();
|
||||
assert_eq!(launches.len(), 2, "{calls}");
|
||||
assert!(launches[0].ends_with("--set expert-vram=0 --set prefill=2048"), "{}", launches[0]);
|
||||
// The M0 kit's normal placement on that PC: 1.5 GiB of experts.
|
||||
assert!(launches[1].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[1]);
|
||||
let probe: Value = serde_json::from_str(&std::fs::read_to_string(l.data.join("config").join("llm-probe.json")).unwrap()).unwrap();
|
||||
assert_eq!(probe["32768"], json!({"context": 32768, "totalMib": 16303, "modelMib": 9874, "stagingGib": 1.1}));
|
||||
llm.stop().await.unwrap();
|
||||
// The next start reuses the measurement: one launch, placed at once (a new server, as the
|
||||
// fake one does not come back after a stop).
|
||||
std::fs::remove_file(l.data.join("calls")).unwrap();
|
||||
let port2 = models_server_until("qwen-test", Duration::from_millis(0), None).await;
|
||||
let s2 = parse_settings(&json!({"llm": {"port": port2}})).unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(20), llm.start(&s2)).await.expect("starts").unwrap();
|
||||
let calls = std::fs::read_to_string(l.data.join("calls")).unwrap();
|
||||
let launches: Vec<&str> = calls.lines().filter(|c| c.starts_with("launch --family")).collect();
|
||||
assert_eq!(launches.len(), 1, "{calls}");
|
||||
assert!(launches[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", launches[0]);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn a_cancel_during_the_download_stops_it_and_clears_the_server() {
|
||||
|
||||
@@ -26,7 +26,7 @@ fn a_launcher_game_turns_game_mode_on_until_it_ends() {
|
||||
eprintln!("skipped: set JARVIS_TEST_GAME_EXE");
|
||||
return;
|
||||
};
|
||||
let w = GameWatch::start(GameSettings { detect: true, games: vec![], not_games: vec![] });
|
||||
let w = GameWatch::start(GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 });
|
||||
assert_eq!(w.view(), serde_json::Value::Null, "no game before it starts");
|
||||
let mut game = std::process::Command::new(&exe).spawn().expect("the game starts");
|
||||
let t0 = Instant::now();
|
||||
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user