feat(app): performance stages move the LLM between the normal and the game placement
- while the services run, game mode decides the stage (perf.rs): a game means the game stage at once (or pause with game.pauseInGame); its end brings the normal stage back after game.revertSeconds (default 30) - a change tells the voice worker the LLM switches (llm switching, with a notice), waits until the brain has written the answers in progress (llm_idle, at most 2 minutes), restarts the LLM in the stage and tells the worker it is ready or paused - the game stage runs at 16K context, one request at a time, half the engine's CPU threads and the game placement (reserve game.reserveGb, default 8, else 2 GiB, else 0 GiB, else pause), measured once at that context; every placement of a run counts other programs' memory as it was when the services started - the voice worker gates every LLM use: on switching new work waits (at most 180 s), the first spoken request that waits hears the notice once under its own id, on ready all go on, on paused nothing is answered - dashboard: "성능 단계: 게임 중 · 게임에 X GB" / 일시정지; game mode screen: game VRAM, revert wait, pause in game Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -192,8 +192,8 @@ reaches the screen.
|
||||
percentage, or `X of Y GB` for the model download; none on a line without progress), also before
|
||||
the services are up; a failure's reason in `note`.
|
||||
- `game`: the game that game mode is on for, `{name, reason}` (below, "Game detection"), or null.
|
||||
- `performance`: the performance stage (normal, gaming, paused, and why) comes next in M7; until
|
||||
then `null`.
|
||||
- `performance`: `{stage, gameRoomMib}` from the LLM: `normal`, `gaming` (with the graphics memory
|
||||
the game gets) or `paused` (below, "Performance stages"); null before the services run.
|
||||
|
||||
The dashboard's state is built from the core's events alone (`Dashboard::on_event` with the time),
|
||||
so it is tested with made-up events.
|
||||
@@ -336,7 +336,34 @@ elsewhere it never sees a game). It never looks at the game's screen.
|
||||
- `game` (new section): `detect` (게임 감지, on by default), `games` (programs that are games, as
|
||||
file names, ".exe" may be left out) and `notGames` (programs that are never games); each list at
|
||||
most 100 names, no folders; `otherRoomGb` (0-8, default 1), the graphics memory the normal
|
||||
placement leaves for programs opened later (above, "LLM"). Read when the services start.
|
||||
placement leaves for programs opened later (above, "LLM"); `reserveGb` (0-32, default 8), the
|
||||
graphics memory a game gets; `revertSeconds` (0-600, default 30), how long after a game ends the
|
||||
normal stage comes back; `pauseInGame` (default off), stop the LLM while a game runs. Read when
|
||||
the services start.
|
||||
|
||||
## Performance stages (`app/src/perf.rs`, `app/src/services.rs`)
|
||||
|
||||
`docs/v2/pc-features.md`, "단계" and "전환 규칙". While the services run, the app moves the LLM
|
||||
between the normal placement and the game placement as games come and go (not with `llm.url`).
|
||||
|
||||
- The stage wanted (checked every second): a game (game mode on) means the game stage at once, or
|
||||
pause with `pauseInGame`; when it ends the normal stage comes back once `revertSeconds` have
|
||||
passed with no game, so quitting and starting a game again does not move the model back and
|
||||
forth.
|
||||
- A change: the brain is told the LLM switches, with a notice for the first spoken request that
|
||||
waits ("잠깐만요, 게임 모드로 바꾸는 중이에요", or "잠깐만요, 평소 모드로 돌아가는 중이에요" on the way
|
||||
back; `voice_worker.spec.md`, "LLM switches"); the app waits until the brain has written the
|
||||
answers in progress (at most 2 minutes), restarts the LLM in the new stage and tells the brain
|
||||
it is ready (or paused). A restart that fails leaves the LLM `failed` on the dashboard and the
|
||||
brain ready (its calls then fail); the stage is not tried again until the wanted stage changes.
|
||||
- The game stage: context 16,384 (or the settings' smaller one), one request at a time, half the
|
||||
engine's CPU threads (read from its log at the normal start, else a quarter of the logical CPUs),
|
||||
and the game placement (`placement.rs`: the reserve beside more than 2 GiB of experts, else
|
||||
2 GiB, else 0 GiB, else pause), measured once at that context like the normal one. Every
|
||||
placement of a run counts other programs' memory as it was when the services started (before
|
||||
the LLM and the STT worker), not from a card that then also holds the worker and the game.
|
||||
- Pause: the LLM stops (`llm` off, "게임 중 일시정지"); the brain answers nothing until the stage
|
||||
changes.
|
||||
- The foreground program is a game, in this order: in `notGames`: no; in `games`: yes
|
||||
(`registered`); a browser (also the bot's Chrome), video player, the desktop, Discord, a
|
||||
launcher's own window or this app: no; under a launcher's game folder (the folder after
|
||||
@@ -568,7 +595,11 @@ the bot's tab, the focus staying in the program the user types into, no taskbar
|
||||
left after the stop, and the uninstall leaving no device, driver package, registry key or settings
|
||||
folder. Game detection (`tests/game_watch.rs`): a program started from a Steam game folder (also under
|
||||
a path with non-ASCII letters) or from a Battle.net game's install folder (a test uninstall entry)
|
||||
turns game mode on with its name and off once it ends. On every system: game detection's rules
|
||||
turns game mode on with its name and off once it ends. Performance stages (on every system, with a fake
|
||||
Strata and card): a game moves the LLM to the game placement (measured once at 16K and one request,
|
||||
2 GiB on the game PC's numbers, half the 16 engine threads) and back after the wait, pause stops
|
||||
it, a card with no room for a game pauses, and the stage loop tells the brain switching (with the
|
||||
notice), waits, restarts and tells it ready, also on the way back. On every system: game detection's rules
|
||||
(a borderless game, a maximised program, a browser's full-screen video, exclusive full screen,
|
||||
launcher folders and their own windows, Battle.net games by install folder and their uninstall
|
||||
entries, non-ASCII paths, the person's lists first, staying on until the game ends,
|
||||
|
||||
@@ -275,7 +275,7 @@ impl Controller {
|
||||
view["system"] = json!(self.system().await);
|
||||
view["llm"] = self.host.llm();
|
||||
view["game"] = status.get("game").cloned().unwrap_or(Value::Null);
|
||||
view["performance"] = Value::Null; // the performance stages come next in M7
|
||||
view["performance"] = status.get("performance").cloned().unwrap_or(Value::Null);
|
||||
view
|
||||
}
|
||||
|
||||
@@ -364,6 +364,7 @@ impl Running for crate::services::Services {
|
||||
"missed": crate::dashboard::missed_view(&missed, now),
|
||||
"browser": self.browser.view(),
|
||||
"game": self.game.view(),
|
||||
"performance": self.llm.stage_view(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -236,6 +236,11 @@ impl GameWatch {
|
||||
Self { state, stop, thread: Mutex::new(thread) }
|
||||
}
|
||||
|
||||
/// Whether game mode is on.
|
||||
pub fn on(&self) -> bool {
|
||||
self.state.lock().unwrap().current().is_some()
|
||||
}
|
||||
|
||||
/// The game for the dashboard: `{name, reason}` or null.
|
||||
pub fn view(&self) -> serde_json::Value {
|
||||
serde_json::json!(self.state.lock().unwrap().current())
|
||||
@@ -479,7 +484,7 @@ mod tests {
|
||||
use super::*;
|
||||
|
||||
fn settings() -> GameSettings {
|
||||
GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 }
|
||||
GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0, reserve_gb: 8.0, revert_seconds: 30, pause_in_game: false }
|
||||
}
|
||||
|
||||
fn fg(path: &str, covers: bool, no_frame: bool, gpu: f64) -> Foreground {
|
||||
|
||||
@@ -21,6 +21,7 @@ pub mod frames;
|
||||
pub mod install;
|
||||
pub mod logs;
|
||||
pub mod playback;
|
||||
pub mod perf;
|
||||
pub mod placement;
|
||||
pub mod policy;
|
||||
pub mod receive;
|
||||
|
||||
79
app/src/perf.rs
Normal file
79
app/src/perf.rs
Normal file
@@ -0,0 +1,79 @@
|
||||
//! The performance stages (`docs/v2/pc-features.md`, "자동 성능 조절" and "전환 규칙"; admin.spec.md,
|
||||
//! "Performance stages"): which placement the LLM should run in, from game mode and time. The
|
||||
//! switching itself (waiting for the answer in progress, restarting the LLM) is `services.rs`.
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use serde::Serialize;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Stage {
|
||||
Normal,
|
||||
Gaming,
|
||||
/// The LLM is stopped while a game runs (chosen, or no placement leaves the game room).
|
||||
Paused,
|
||||
}
|
||||
|
||||
/// Picks the stage: a game means the game stage (or pause when the person chose it) at once; after
|
||||
/// the game ends the normal stage comes back only once `revert_after` has passed, so quitting and
|
||||
/// starting a game again does not move the model back and forth.
|
||||
#[derive(Debug)]
|
||||
pub struct Perf {
|
||||
revert_after: Duration,
|
||||
pause_in_game: bool,
|
||||
game_since_off: Option<Instant>,
|
||||
current: Stage,
|
||||
}
|
||||
|
||||
impl Perf {
|
||||
pub fn new(revert_after: Duration, pause_in_game: bool) -> Self {
|
||||
Self { revert_after, pause_in_game, game_since_off: None, current: Stage::Normal }
|
||||
}
|
||||
|
||||
/// The stage wanted now, given whether a game runs.
|
||||
pub fn target(&mut self, game: bool, now: Instant) -> Stage {
|
||||
if game {
|
||||
self.game_since_off = None;
|
||||
self.current = if self.pause_in_game { Stage::Paused } else { Stage::Gaming };
|
||||
} else if self.current != Stage::Normal {
|
||||
let since = *self.game_since_off.get_or_insert(now);
|
||||
if now.duration_since(since) >= self.revert_after {
|
||||
self.current = Stage::Normal;
|
||||
self.game_since_off = None;
|
||||
}
|
||||
}
|
||||
self.current
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_game_switches_at_once_and_its_end_only_after_the_wait() {
|
||||
let t = Instant::now();
|
||||
let s = |n: u64| t + Duration::from_secs(n);
|
||||
let mut p = Perf::new(Duration::from_secs(30), false);
|
||||
assert_eq!(p.target(false, s(0)), Stage::Normal);
|
||||
assert_eq!(p.target(true, s(1)), Stage::Gaming);
|
||||
// The game ends: still the game stage for 30 s.
|
||||
assert_eq!(p.target(false, s(2)), Stage::Gaming);
|
||||
assert_eq!(p.target(false, s(31)), Stage::Gaming);
|
||||
// Started again within the wait: nothing moves, and the wait starts over at the next end.
|
||||
assert_eq!(p.target(true, s(32)), Stage::Gaming);
|
||||
assert_eq!(p.target(false, s(40)), Stage::Gaming);
|
||||
assert_eq!(p.target(false, s(69)), Stage::Gaming);
|
||||
assert_eq!(p.target(false, s(70)), Stage::Normal);
|
||||
assert_eq!(p.target(false, s(500)), Stage::Normal);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn with_pause_chosen_a_game_pauses_the_llm() {
|
||||
let t = Instant::now();
|
||||
let mut p = Perf::new(Duration::ZERO, true);
|
||||
assert_eq!(p.target(true, t), Stage::Paused);
|
||||
assert_eq!(p.target(false, t), Stage::Normal);
|
||||
}
|
||||
}
|
||||
@@ -24,7 +24,9 @@ pub struct Services {
|
||||
pub llm: Arc<Llm>,
|
||||
pub browser: BrowserRun,
|
||||
/// Game detection while the services run (admin.spec.md, "Game detection").
|
||||
pub game: crate::game::GameWatch,
|
||||
pub game: Arc<crate::game::GameWatch>,
|
||||
/// Moves the LLM between the performance stages as games come and go.
|
||||
pub perf: std::sync::Mutex<Option<tokio::task::JoinHandle<()>>>,
|
||||
}
|
||||
|
||||
/// The bot browser of a run (admin.spec.md, "Bot browser").
|
||||
@@ -61,6 +63,11 @@ impl BrowserRun {
|
||||
impl Services {
|
||||
/// Stop everything; an error means the LLM's server may still run.
|
||||
pub async fn stop(&self) -> Result<(), String> {
|
||||
let perf = self.perf.lock().unwrap().take();
|
||||
if let Some(p) = perf {
|
||||
p.abort();
|
||||
let _ = p.await;
|
||||
}
|
||||
self.game.stop();
|
||||
self.core.close().await;
|
||||
self.worker.close();
|
||||
@@ -113,6 +120,7 @@ async fn start_browser(data_dir: &Path, settings: &crate::settings::Settings) ->
|
||||
pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let settings = load_settings(&data_dir.join("config").join("settings.json"))?;
|
||||
let game_settings = settings.game.clone();
|
||||
let perf_settings = settings.clone();
|
||||
// The stored token (the setup wizard keeps it encrypted, admin.spec.md), else the environment.
|
||||
let secrets = crate::secrets::Secrets::new(&data_dir.join("config").join("secrets.dat"));
|
||||
let stored = secrets.get(crate::secrets::DISCORD_TOKEN)?;
|
||||
@@ -193,8 +201,76 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
|
||||
return Err(after_stop(&llm, &browser, e.to_string()).await.into());
|
||||
}
|
||||
crate::info_log!("app", "services started");
|
||||
let game = crate::game::GameWatch::start(game_settings);
|
||||
Ok(Services { core, events, worker, worker_process, missed, llm, browser, game })
|
||||
let game = Arc::new(crate::game::GameWatch::start(game_settings));
|
||||
let on = { let g = game.clone(); Arc::new(move || g.on()) };
|
||||
let perf = std::sync::Mutex::new(Some(tokio::spawn(perf_loop(perf_settings, llm.clone(), Arc::new(worker.clone()), on))));
|
||||
Ok(Services { core, events, worker, worker_process, missed, llm, browser, game, perf })
|
||||
}
|
||||
|
||||
/// The brain as the performance stages see it: told the LLM's state, and asked to finish what it is
|
||||
/// answering (the worker client; a fake in tests).
|
||||
#[async_trait::async_trait]
|
||||
pub trait LlmBrain: Send + Sync {
|
||||
fn llm_state(&self, state: &str, notice: Option<&str>) -> bool;
|
||||
async fn wait_llm_idle(&self, t: std::time::Duration) -> bool;
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LlmBrain for WorkerClient {
|
||||
fn llm_state(&self, state: &str, notice: Option<&str>) -> bool {
|
||||
WorkerClient::llm_state(self, state, notice)
|
||||
}
|
||||
async fn wait_llm_idle(&self, t: std::time::Duration) -> bool {
|
||||
WorkerClient::wait_llm_idle(self, t).await
|
||||
}
|
||||
}
|
||||
|
||||
/// What a request that waits for a switch hears once (voice_worker.spec.md, "LLM switches").
|
||||
fn switch_notice(stage: crate::perf::Stage) -> Option<&'static str> {
|
||||
match stage {
|
||||
crate::perf::Stage::Gaming => Some("잠깐만요, 게임 모드로 바꾸는 중이에요"),
|
||||
crate::perf::Stage::Normal => Some("잠깐만요, 평소 모드로 돌아가는 중이에요"),
|
||||
crate::perf::Stage::Paused => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// The performance stages (admin.spec.md, "Performance stages"): every second the stage wanted
|
||||
/// from game mode (`perf.rs`); a change tells the brain the LLM switches, waits until the answers
|
||||
/// in progress are written (at most 2 minutes), restarts the LLM in that stage (or stops it to
|
||||
/// pause) and tells the brain it is ready (or paused). With `llm.url` nothing moves.
|
||||
async fn perf_loop(settings: crate::settings::Settings, llm: Arc<Llm>, worker: Arc<dyn LlmBrain>, game: Arc<dyn Fn() -> bool + Send + Sync>) {
|
||||
use crate::perf::{Perf, Stage};
|
||||
if settings.llm.url.is_some() {
|
||||
return;
|
||||
}
|
||||
let mut perf = Perf::new(std::time::Duration::from_secs(settings.game.revert_seconds.into()), settings.game.pause_in_game);
|
||||
let mut applied = Stage::Normal;
|
||||
loop {
|
||||
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
|
||||
let want = perf.target(game(), std::time::Instant::now());
|
||||
if want == applied {
|
||||
continue;
|
||||
}
|
||||
crate::info_log!("perf", "performance stage: {want:?}");
|
||||
worker.llm_state("switching", switch_notice(want));
|
||||
if !worker.wait_llm_idle(std::time::Duration::from_secs(120)).await {
|
||||
crate::warn_log!("perf", "Warning: the brain did not finish its answers in 2 minutes; switching anyway");
|
||||
}
|
||||
// Wanted is what counts as applied: a game stage that ends paused is not tried again.
|
||||
applied = want;
|
||||
match llm.switch(&settings, want).await {
|
||||
Ok(Stage::Paused) => {
|
||||
worker.llm_state("paused", None);
|
||||
}
|
||||
Ok(_) => {
|
||||
worker.llm_state("ready", None);
|
||||
}
|
||||
Err(e) => {
|
||||
crate::error_log!("perf", "the LLM did not come back in the {want:?} stage: {e}");
|
||||
worker.llm_state("ready", None);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The secret values the services get: the bot token and the MCP servers' environment values.
|
||||
@@ -220,6 +296,70 @@ async fn after_llm_stop(llm: &Llm, error: String) -> String {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod perf_tests {
|
||||
use super::*;
|
||||
use crate::strata::tests::{card, fake_placed, launches, layout, server_after_launch};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Mutex;
|
||||
use std::time::Duration;
|
||||
|
||||
#[derive(Default)]
|
||||
struct FakeBrain(Mutex<Vec<String>>);
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LlmBrain for FakeBrain {
|
||||
fn llm_state(&self, state: &str, notice: Option<&str>) -> bool {
|
||||
self.0.lock().unwrap().push(format!("{state}{}", notice.map(|n| format!(": {n}")).unwrap_or_default()));
|
||||
true
|
||||
}
|
||||
async fn wait_llm_idle(&self, _t: Duration) -> bool {
|
||||
self.0.lock().unwrap().push("waited".into());
|
||||
true
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn a_game_moves_the_llm_to_the_game_stage_and_its_end_moves_it_back() {
|
||||
let l = layout("perf-loop");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let llm = Arc::new(Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(card(&l, "GPU-test", 0)));
|
||||
let s = crate::settings::parse_settings(&serde_json::json!({"llm": {"port": port}, "game": {"revertSeconds": 0}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
let brain = Arc::new(FakeBrain::default());
|
||||
let game = Arc::new(AtomicBool::new(false));
|
||||
let g = game.clone();
|
||||
let task = tokio::spawn(perf_loop(s, llm.clone(), brain.clone(), Arc::new(move || g.load(Ordering::SeqCst))));
|
||||
std::fs::remove_file(l.data.join("calls")).unwrap();
|
||||
game.store(true, Ordering::SeqCst);
|
||||
let until = |want: &'static str| {
|
||||
let brain = brain.clone();
|
||||
async move {
|
||||
for _ in 0..200 {
|
||||
if brain.0.lock().unwrap().iter().filter(|m| m.as_str() == want).count() > 0 {
|
||||
return;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
panic!("never {want}: {:?}", brain.0.lock().unwrap());
|
||||
}
|
||||
};
|
||||
until("ready").await;
|
||||
assert_eq!(*brain.0.lock().unwrap(), ["switching: 잠깐만요, 게임 모드로 바꾸는 중이에요", "waited", "ready"]);
|
||||
assert!(launches(&l).last().unwrap().contains("--context 16384"), "{:?}", launches(&l));
|
||||
assert_eq!(llm.stage_view()["stage"], serde_json::json!("gaming"));
|
||||
// The game ends (no wait set): back to the normal stage.
|
||||
brain.0.lock().unwrap().clear();
|
||||
game.store(false, Ordering::SeqCst);
|
||||
until("ready").await;
|
||||
assert_eq!(brain.0.lock().unwrap()[0], "switching: 잠깐만요, 평소 모드로 돌아가는 중이에요");
|
||||
assert!(launches(&l).last().unwrap().contains("--context 32768"));
|
||||
assert_eq!(llm.stage_view()["stage"], serde_json::json!("normal"));
|
||||
task.abort();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -109,12 +109,22 @@ pub struct GameSettings {
|
||||
pub not_games: Vec<String>,
|
||||
/// Graphics memory the normal placement leaves for programs opened later (GB, default 1).
|
||||
pub other_room_gb: f64,
|
||||
/// Graphics memory a game gets in the game stage (GB, default 8).
|
||||
pub reserve_gb: f64,
|
||||
/// How long after a game ends the normal stage comes back (seconds, default 30).
|
||||
pub revert_seconds: u32,
|
||||
/// Stop the LLM while a game runs (default off).
|
||||
pub pause_in_game: bool,
|
||||
}
|
||||
|
||||
impl GameSettings {
|
||||
pub fn other_room_mib(&self) -> i64 {
|
||||
(self.other_room_gb * 1024.0).round() as i64
|
||||
}
|
||||
|
||||
pub fn reserve_mib(&self) -> i64 {
|
||||
(self.reserve_gb * 1024.0).round() as i64
|
||||
}
|
||||
}
|
||||
|
||||
/// Background collection (admin.spec.md, "Background collection"); no targets by default.
|
||||
@@ -210,7 +220,7 @@ impl Default for Settings {
|
||||
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true, browser_share: false },
|
||||
mcp_servers: vec![],
|
||||
collect: CollectSettings { keep_days: 30, targets: vec![] },
|
||||
game: GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 },
|
||||
game: GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0, reserve_gb: 8.0, revert_seconds: 30, pause_in_game: false },
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -571,15 +581,21 @@ fn parse_game(raw: &Map<String, Value>, d: &GameSettings) -> Result<GameSettings
|
||||
}
|
||||
Ok(v)
|
||||
};
|
||||
let other_room_gb = match g.and_then(|m| m.get("otherRoomGb")) {
|
||||
None => d.other_room_gb,
|
||||
Some(v) => match v.as_f64() {
|
||||
Some(n) if (0.0..=8.0).contains(&n) => n,
|
||||
_ => return err("game.otherRoomGb must be a number 0-8 (GB)"),
|
||||
},
|
||||
let gb = |key: &str, default: f64, max: f64| -> Result<f64, SettingsError> {
|
||||
match g.and_then(|m| m.get(key)) {
|
||||
None => Ok(default),
|
||||
Some(v) => match v.as_f64() {
|
||||
Some(n) if (0.0..=max).contains(&n) => Ok(n),
|
||||
_ => err(format!("game.{key} must be a number 0-{max} (GB)")),
|
||||
},
|
||||
}
|
||||
};
|
||||
let other_room_gb = gb("otherRoomGb", d.other_room_gb, 8.0)?;
|
||||
let reserve_gb = gb("reserveGb", d.reserve_gb, 32.0)?;
|
||||
let revert_seconds = get_uint(g, "revertSeconds", "game", d.revert_seconds, 0, 600)?;
|
||||
Ok(GameSettings { detect: get_bool(g, "detect", "game", d.detect)?, games: list("games", &d.games)?,
|
||||
not_games: list("notGames", &d.not_games)?, other_room_gb })
|
||||
not_games: list("notGames", &d.not_games)?, other_room_gb, reserve_gb, revert_seconds,
|
||||
pause_in_game: get_bool(g, "pauseInGame", "game", d.pause_in_game)? })
|
||||
}
|
||||
|
||||
fn parse_collect(raw: &Map<String, Value>, d: &CollectSettings) -> Result<CollectSettings, SettingsError> {
|
||||
@@ -851,11 +867,13 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn game_detection_settings_are_read_and_checked() {
|
||||
assert_eq!(parse_settings(&json!({})).unwrap().game, GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 });
|
||||
let s = parse_settings(&json!({"game": {"detect": false, "games": [" MyGame.exe ", ""], "notGames": ["obs64"], "otherRoomGb": 2.5}})).unwrap();
|
||||
assert_eq!(s.game, GameSettings { detect: false, games: vec!["MyGame.exe".into()], not_games: vec!["obs64".into()], other_room_gb: 2.5 });
|
||||
assert_eq!(parse_settings(&json!({})).unwrap().game, GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0, reserve_gb: 8.0, revert_seconds: 30, pause_in_game: false });
|
||||
let s = parse_settings(&json!({"game": {"detect": false, "games": [" MyGame.exe ", ""], "notGames": ["obs64"], "otherRoomGb": 2.5, "reserveGb": 6, "revertSeconds": 60, "pauseInGame": true}})).unwrap();
|
||||
assert_eq!(s.game, GameSettings { detect: false, games: vec!["MyGame.exe".into()], not_games: vec!["obs64".into()], other_room_gb: 2.5,
|
||||
reserve_gb: 6.0, revert_seconds: 60, pause_in_game: true });
|
||||
assert_eq!(s.game.reserve_mib(), 6144);
|
||||
assert_eq!(s.game.other_room_mib(), 2560);
|
||||
for bad in [json!({"game": {"games": ["C:\\x\\a.exe"]}}), json!({"game": {"notGames": "a"}}), json!({"game": {"detect": 1}}), json!({"game": {"otherRoomGb": 9}})] {
|
||||
for bad in [json!({"game": {"games": ["C:\\x\\a.exe"]}}), json!({"game": {"notGames": "a"}}), json!({"game": {"detect": 1}}), json!({"game": {"otherRoomGb": 9}}), json!({"game": {"revertSeconds": 601}}), json!({"game": {"reserveGb": -1}})] {
|
||||
assert!(parse_settings(&bad).unwrap_err().0.contains("game."), "{bad}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -134,6 +134,9 @@ pub fn fields() -> Vec<Field> {
|
||||
f("game.detect", Game, Bool, json!(true), "게임 감지"),
|
||||
f("game.games", Game, TextList { max_items: 100, nullable: false }, json!([]), "게임으로 볼 프로그램(파일 이름, 예: MyGame.exe)"),
|
||||
f("game.otherRoomGb", Game, Number { min: 0.0, max: 8.0 }, json!(1.0), "평소 다른 프로그램용으로 남길 VRAM(GB)"),
|
||||
f("game.reserveGb", Game, Number { min: 0.0, max: 32.0 }, json!(8.0), "게임 중 게임용으로 비울 VRAM(GB)"),
|
||||
f("game.revertSeconds", Game, Int { min: 0, max: 600 }, json!(30), "게임이 끝난 뒤 평소 배치로 돌아가기까지(초)"),
|
||||
f("game.pauseInGame", Game, Bool, json!(false), "게임 중 일시정지(대답하지 않음)"),
|
||||
f("game.notGames", Game, TextList { max_items: 100, nullable: false }, json!([]), "게임이 아닌 프로그램(파일 이름)"),
|
||||
]
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ use serde::Serialize;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
use crate::install::{Ctx, Layout};
|
||||
use crate::perf::Stage;
|
||||
use crate::settings::Settings;
|
||||
|
||||
/// The one model v2 runs (M0: Qwen3.6-35B-A3B UD-Q4_K_XL; docs/v2/roadmap.md).
|
||||
@@ -49,12 +50,12 @@ pub fn parallel(settings: &Settings) -> u32 {
|
||||
/// `strata launch` for the services: the M0 kit's normal placement, models in `models/llm`, this
|
||||
/// computer only. No pictures yet (nothing in v2 sends one; reading them takes graphics memory).
|
||||
/// `--restart` replaces a server started with other choices; the same choices are reused.
|
||||
pub fn launch_args(layout: &Layout, settings: &Settings, sets: &[String]) -> Vec<String> {
|
||||
pub fn launch_args(layout: &Layout, settings: &Settings, conf: &Conf, sets: &[String]) -> Vec<String> {
|
||||
let models = layout.data.join("models").join("llm").to_string_lossy().into_owned();
|
||||
let mut a: Vec<String> = ["launch", "--family", FAMILY, "--model", MODEL, "--data-dir"].map(String::from).into();
|
||||
a.push(models);
|
||||
for (k, v) in [("--context", settings.llm.context.to_string()), ("--vision", "off".into()),
|
||||
("--parallel", parallel(settings).to_string()), ("--access", "local".into()),
|
||||
for (k, v) in [("--context", conf.context.to_string()), ("--vision", "off".into()),
|
||||
("--parallel", conf.parallel.to_string()), ("--access", "local".into()),
|
||||
("--port", settings.llm.port.to_string())] {
|
||||
a.push(k.into());
|
||||
a.push(v);
|
||||
@@ -67,6 +68,29 @@ pub fn launch_args(layout: &Layout, settings: &Settings, sets: &[String]) -> Vec
|
||||
a
|
||||
}
|
||||
|
||||
/// The context and requests at once of a stage: the settings' in the normal stage; in a game 16K
|
||||
/// (or less when the settings ask for less) and one request at a time.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct Conf {
|
||||
pub context: u32,
|
||||
pub parallel: u32,
|
||||
}
|
||||
|
||||
pub fn conf(settings: &Settings, stage: Stage) -> Conf {
|
||||
match stage {
|
||||
Stage::Gaming => Conf { context: crate::placement::GAME_CONTEXT.min(settings.llm.context), parallel: 1 },
|
||||
_ => Conf { context: settings.llm.context, parallel: parallel(settings) },
|
||||
}
|
||||
}
|
||||
|
||||
/// The engine's CPU threads from its log ("... on 16 CPU threads ...").
|
||||
fn cpu_threads(log: &str) -> Option<u32> {
|
||||
log.lines().rev().find_map(|l| {
|
||||
let i = l.find(" CPU threads")?;
|
||||
l[..i].rsplit(' ').next()?.parse().ok()
|
||||
})
|
||||
}
|
||||
|
||||
/// The engine settings for an expert budget (`placement.rs`) and the M0 kit's prompt chunk.
|
||||
fn placement_sets(expert_gib: f64) -> Vec<String> {
|
||||
vec![format!("expert-vram={expert_gib}"), "prefill=2048".into()]
|
||||
@@ -228,6 +252,14 @@ pub struct Llm {
|
||||
stop_timeout: Duration,
|
||||
/// The graphics card's memory (none: Strata places the model itself).
|
||||
gpu: Option<GpuReader>,
|
||||
/// What other programs held when the services started (before the LLM and the STT worker):
|
||||
/// every later placement of this run counts from it, not from a card that then also holds the
|
||||
/// STT worker and a game.
|
||||
baseline: Mutex<Option<i64>>,
|
||||
/// The engine's CPU threads in the normal stage; a game gets half.
|
||||
threads: Mutex<Option<u32>>,
|
||||
/// The stage it runs in, and in a game the graphics memory the game gets.
|
||||
stage: Mutex<(Stage, Option<i64>)>,
|
||||
}
|
||||
|
||||
/// How long the model may take to answer after `strata launch` returned (it loads in about 15 s
|
||||
@@ -247,7 +279,14 @@ impl Llm {
|
||||
pub fn with_exe(layout: Layout, exe: std::path::PathBuf) -> Self {
|
||||
Self { layout, exe, status: Mutex::new(LlmStatus { state: LlmState::Off, managed: false, percent: None, note: None }),
|
||||
active: Mutex::new(None), cancel: Arc::new(AtomicBool::new(false)), port: Mutex::new(0),
|
||||
stop_timeout: Duration::from_secs(120), gpu: None }
|
||||
stop_timeout: Duration::from_secs(120), gpu: None, baseline: Mutex::new(None), threads: Mutex::new(None),
|
||||
stage: Mutex::new((Stage::Normal, None)) }
|
||||
}
|
||||
|
||||
/// `{stage, gameRoomMib}` for the dashboard.
|
||||
pub fn stage_view(&self) -> Value {
|
||||
let (stage, room) = *self.stage.lock().unwrap();
|
||||
json!({"stage": stage, "gameRoomMib": room})
|
||||
}
|
||||
|
||||
/// Place the model by the card's memory (`placement.rs`), read with `gpu`.
|
||||
@@ -272,19 +311,29 @@ impl Llm {
|
||||
/// counted as another program's memory (a stop that is not confirmed reads nothing: no
|
||||
/// experts). A measurement that fails is not kept: the model then starts without experts on
|
||||
/// the card.
|
||||
async fn placement(&self, settings: &Settings, ctx: &Ctx, base: &str) -> Result<Vec<String>, String> {
|
||||
async fn placement(&self, settings: &Settings, ctx: &Ctx, base: &str, stage: Stage) -> Result<Option<Vec<String>>, String> {
|
||||
let c = conf(settings, stage);
|
||||
let workers = |sets: &mut Vec<String>| if stage == Stage::Gaming {
|
||||
let half = self.threads.lock().unwrap().map(|n| n / 2)
|
||||
.unwrap_or_else(|| std::thread::available_parallelism().map(|n| n.get() as u32 / 4).unwrap_or(1));
|
||||
sets.push(format!("pool-workers={}", half.max(1)));
|
||||
};
|
||||
if self.gpu.is_none() {
|
||||
return Ok(vec![]);
|
||||
let mut sets = vec![];
|
||||
workers(&mut sets);
|
||||
return Ok(Some(sets));
|
||||
}
|
||||
// Whatever a server left from an earlier run does (answers, loads, stays silent), it holds
|
||||
// the card: stop it first and read the card only once the port is confirmed closed.
|
||||
if let Err(e) = self.stop_server_on(settings.llm.port).await {
|
||||
crate::warn_log!("llm", "Warning: a server left on the port did not stop ({e}); starting without experts on the card");
|
||||
return Ok(placement_sets(0.0));
|
||||
let mut sets = placement_sets(0.0);
|
||||
workers(&mut sets);
|
||||
return Ok(Some(sets));
|
||||
}
|
||||
let Some(card) = self.read_gpu().await else { return Ok(vec![]) };
|
||||
let (context, parallel) = (settings.llm.context, parallel(settings));
|
||||
let key = format!("{context}/{parallel}");
|
||||
let Some(card) = self.read_gpu().await else { return Ok(Some(vec![])) };
|
||||
let others = *self.baseline.lock().unwrap().get_or_insert(card.used_mib);
|
||||
let key = format!("{}/{}", c.context, c.parallel);
|
||||
let file = self.probe_file();
|
||||
let mut all: serde_json::Map<String, Value> = std::fs::read_to_string(&file).ok()
|
||||
.and_then(|t| serde_json::from_str(&t).ok()).unwrap_or_default();
|
||||
@@ -293,16 +342,18 @@ impl Llm {
|
||||
let probe = match known {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
crate::info_log!("llm", "measuring the model without experts once (context {context}, {parallel} at once)");
|
||||
self.launch(settings, ctx, base, &placement_sets(0.0)).await?;
|
||||
crate::info_log!("llm", "measuring the model without experts once (context {}, {} at once)", c.context, c.parallel);
|
||||
self.launch(settings, ctx, base, &c, &placement_sets(0.0)).await?;
|
||||
let warmed = warm_up(base).await;
|
||||
let after = self.read_gpu().await.filter(|g| g.id == card.id);
|
||||
let model_mib = after.map(|g| g.used_mib - card.used_mib).unwrap_or(0);
|
||||
if let Err(e) = warmed.and_then(|()| if model_mib > 0 { Ok(()) } else { Err("the card's memory could not be read".into()) }) {
|
||||
crate::warn_log!("llm", "Warning: the measurement failed ({e}); starting without experts on the card");
|
||||
return Ok(placement_sets(0.0));
|
||||
let mut sets = placement_sets(0.0);
|
||||
workers(&mut sets);
|
||||
return Ok(Some(sets));
|
||||
}
|
||||
let p = crate::placement::Probe { context, parallel, gpu: card.id.clone(), model_mib,
|
||||
let p = crate::placement::Probe { context: c.context, parallel: c.parallel, gpu: card.id.clone(), model_mib,
|
||||
staging_gib: crate::placement::staging_gib(&engine_log(&self.layout.strata_home())) };
|
||||
all.insert(key, json!(p));
|
||||
crate::datadir::write_atomic(&file, Value::Object(all).to_string().as_bytes())
|
||||
@@ -310,15 +361,28 @@ impl Llm {
|
||||
p
|
||||
}
|
||||
};
|
||||
let expert = crate::placement::normal(card.total_mib, card.used_mib, &probe, settings.game.other_room_mib());
|
||||
crate::info_log!("llm", "placement: experts {expert} GiB (card {} MiB, held by others {} MiB, model without experts {} MiB)",
|
||||
card.total_mib, card.used_mib, probe.model_mib);
|
||||
Ok(placement_sets(expert))
|
||||
let expert = match stage {
|
||||
Stage::Gaming => {
|
||||
let g = crate::placement::game(card.total_mib, others, &probe, settings.game.reserve_mib());
|
||||
if g.paused {
|
||||
crate::info_log!("llm", "placement: no room for a game beside the model; paused while the game runs");
|
||||
return Ok(None);
|
||||
}
|
||||
*self.stage.lock().unwrap() = (Stage::Gaming, Some(g.game_room_mib));
|
||||
g.expert_gib
|
||||
}
|
||||
_ => crate::placement::normal(card.total_mib, others, &probe, settings.game.other_room_mib()),
|
||||
};
|
||||
crate::info_log!("llm", "placement ({stage:?}): experts {expert} GiB (card {} MiB, held by others {others} MiB, model without experts {} MiB)",
|
||||
card.total_mib, probe.model_mib);
|
||||
let mut sets = placement_sets(expert);
|
||||
workers(&mut sets);
|
||||
Ok(Some(sets))
|
||||
}
|
||||
|
||||
/// `strata launch` with `sets`, then wait until the model answers.
|
||||
async fn launch(&self, settings: &Settings, ctx: &Ctx, base: &str, sets: &[String]) -> Result<String, String> {
|
||||
let args = launch_args(&self.layout, settings, sets);
|
||||
async fn launch(&self, settings: &Settings, ctx: &Ctx, base: &str, c: &Conf, sets: &[String]) -> Result<String, String> {
|
||||
let args = launch_args(&self.layout, settings, c, sets);
|
||||
let refs: Vec<&str> = args.iter().map(String::as_str).collect();
|
||||
ctx.run_in(&self.exe, &refs, Some(&self.layout.strata_home())).await?;
|
||||
wait_ready(base, READY_TIMEOUT, &self.cancel).await
|
||||
@@ -351,11 +415,47 @@ impl Llm {
|
||||
/// Start (or reuse) the LLM; returns its `/v1` address and model id.
|
||||
/// A cancel since the last `arm` (or before the first start) ends it before it runs anything.
|
||||
pub async fn start(&self, settings: &Settings) -> Result<(String, String), String> {
|
||||
// A new run: other programs' memory is read afresh.
|
||||
*self.baseline.lock().unwrap() = None;
|
||||
*self.stage.lock().unwrap() = (Stage::Normal, None);
|
||||
self.start_stage(settings, Stage::Normal).await.map(|r| r.expect("the normal stage never pauses"))
|
||||
}
|
||||
|
||||
/// Move the running LLM to `stage` (`perf.rs`): a restart with that stage's placement, or a stop
|
||||
/// when it pauses (also a game stage with no room for the game). Returns the stage it ended in.
|
||||
/// With `llm.url` nothing moves.
|
||||
pub async fn switch(&self, settings: &Settings, stage: Stage) -> Result<Stage, String> {
|
||||
if settings.llm.url.is_some() {
|
||||
return Ok(Stage::Normal);
|
||||
}
|
||||
if stage != Stage::Paused && self.start_stage(settings, stage).await?.is_some() {
|
||||
return Ok(stage);
|
||||
}
|
||||
*self.port.lock().unwrap() = settings.llm.port;
|
||||
self.stop().await?;
|
||||
*self.stage.lock().unwrap() = (Stage::Paused, None);
|
||||
self.set(LlmState::Off, Some("게임 중 일시정지".into()));
|
||||
Ok(Stage::Paused)
|
||||
}
|
||||
|
||||
async fn start_stage(&self, settings: &Settings, stage: Stage) -> Result<Option<(String, String)>, String> {
|
||||
let managed = settings.llm.url.is_none();
|
||||
*self.status.lock().unwrap() = LlmStatus { state: LlmState::Starting, managed, percent: None, note: None };
|
||||
let result = self.start_inner(settings).await;
|
||||
let result = self.start_inner(settings, stage).await;
|
||||
if stage == Stage::Normal && result.as_ref().is_ok_and(Option::is_some) && self.gpu.is_some() {
|
||||
if let Some(n) = cpu_threads(&engine_log(&self.layout.strata_home())) {
|
||||
*self.threads.lock().unwrap() = Some(n);
|
||||
}
|
||||
}
|
||||
if let Ok(Some(_)) = &result {
|
||||
let mut st = self.stage.lock().unwrap();
|
||||
if stage == Stage::Normal {
|
||||
*st = (Stage::Normal, None);
|
||||
}
|
||||
}
|
||||
match &result {
|
||||
Ok((base, model)) => {
|
||||
Ok(None) => {}
|
||||
Ok(Some((base, model))) => {
|
||||
crate::info_log!("llm", "ready: {base} model={model}");
|
||||
self.set(LlmState::Ready, None);
|
||||
}
|
||||
@@ -368,13 +468,14 @@ impl Llm {
|
||||
result
|
||||
}
|
||||
|
||||
async fn start_inner(&self, settings: &Settings) -> Result<(String, String), String> {
|
||||
/// None: the stage pauses (nothing started).
|
||||
async fn start_inner(&self, settings: &Settings, stage: Stage) -> Result<Option<(String, String)>, String> {
|
||||
if self.cancel.load(Ordering::SeqCst) {
|
||||
return Err("cancelled".into());
|
||||
}
|
||||
if let Some(url) = &settings.llm.url {
|
||||
let model = wait_ready(url, URL_TIMEOUT, &self.cancel).await?;
|
||||
return Ok((url.clone(), model));
|
||||
return Ok(Some((url.clone(), model)));
|
||||
}
|
||||
if !self.exe.exists() {
|
||||
return Err("Strata가 설치되어 있지 않습니다. 마법사의 모델 단계에서 설치해 주세요".into());
|
||||
@@ -383,13 +484,16 @@ impl Llm {
|
||||
*self.active.lock().unwrap() = Some(ctx.clone());
|
||||
*self.port.lock().unwrap() = settings.llm.port;
|
||||
let base = format!("http://127.0.0.1:{}/v1", settings.llm.port);
|
||||
let ready = match self.placement(settings, &ctx, &base).await {
|
||||
Ok(sets) => self.launch(settings, &ctx, &base, &sets).await,
|
||||
let c = conf(settings, stage);
|
||||
let ready = match self.placement(settings, &ctx, &base, stage).await {
|
||||
Ok(Some(sets)) => self.launch(settings, &ctx, &base, &c, &sets).await.map(Some),
|
||||
Ok(None) => Ok(None),
|
||||
Err(e) => Err(e),
|
||||
};
|
||||
*self.active.lock().unwrap() = None;
|
||||
match ready {
|
||||
Ok(model) => Ok((base, model)),
|
||||
Ok(Some(model)) => Ok(Some((base, model))),
|
||||
Ok(None) => Ok(None),
|
||||
// Nothing half-started stays on the graphics card; if it may have, say so.
|
||||
Err(e) => match self.stop_server().await {
|
||||
Ok(()) => Err(e),
|
||||
@@ -491,12 +595,12 @@ async fn wait_gone(base: &str, timeout: Duration) -> bool {
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
pub(crate) mod tests {
|
||||
use super::*;
|
||||
use crate::datadir::temp_dir;
|
||||
use crate::settings::{parse_settings, ChatChannelSettings};
|
||||
|
||||
fn layout(tag: &str) -> Layout {
|
||||
pub(crate) fn layout(tag: &str) -> Layout {
|
||||
Layout { data: temp_dir(tag), src: std::path::PathBuf::from("src") }
|
||||
}
|
||||
|
||||
@@ -504,7 +608,7 @@ mod tests {
|
||||
fn the_launch_keeps_the_model_in_the_data_folder_and_follows_the_settings() {
|
||||
let l = layout("llm1");
|
||||
let s = parse_settings(&json!({"llm": {"context": 16384, "port": 8099}})).unwrap();
|
||||
let a = launch_args(&l, &s, &[]);
|
||||
let a = launch_args(&l, &s, &conf(&s, Stage::Normal), &[]);
|
||||
let after = |k: &str| a[a.iter().position(|x| x == k).unwrap() + 1].clone();
|
||||
assert!(Path::new(&after("--data-dir")).starts_with(&l.data));
|
||||
assert_eq!((after("--context"), after("--port"), after("--parallel")), ("16384".into(), "8099".into(), "1".into()));
|
||||
@@ -566,7 +670,7 @@ mod tests {
|
||||
|
||||
/// `mode`: "list" answers, "hang" keeps the connection and says nothing, "503" is a server still
|
||||
/// loading.
|
||||
async fn serve(id: &'static str, delay: Duration, gone: Option<std::path::PathBuf>, mode: Arc<Mutex<&'static str>>) -> u16 {
|
||||
pub(crate) async fn serve(id: &'static str, delay: Duration, gone: Option<std::path::PathBuf>, mode: Arc<Mutex<&'static str>>) -> u16 {
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
@@ -640,12 +744,12 @@ mod tests {
|
||||
/// A fake `strata`: `launch --stop` records itself and runs `stop_body` (by default it stops the
|
||||
/// server: the file `gone` appears); `launch ...` runs `launch_body`.
|
||||
#[cfg(unix)]
|
||||
fn fake_strata(l: &Layout, launch_body: &str) -> std::path::PathBuf {
|
||||
pub(crate) fn fake_strata(l: &Layout, launch_body: &str) -> std::path::PathBuf {
|
||||
fake_strata_stop(l, launch_body, &format!("touch '{}'", l.data.join("gone").display()))
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn fake_strata_stop(l: &Layout, launch_body: &str, stop_body: &str) -> std::path::PathBuf {
|
||||
pub(crate) fn fake_strata_stop(l: &Layout, launch_body: &str, stop_body: &str) -> std::path::PathBuf {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let exe = l.data.join("fake-strata");
|
||||
std::fs::create_dir_all(l.strata_home()).unwrap();
|
||||
@@ -734,7 +838,7 @@ mod tests {
|
||||
/// server still left over, `leftover` MiB, until a `launch --stop`), and 9874 MiB more while a
|
||||
/// Strata server runs (the calls file shows the last launch and stop).
|
||||
#[cfg(unix)]
|
||||
fn card(l: &Layout, id: &'static str, leftover: i64) -> GpuReader {
|
||||
pub(crate) fn card(l: &Layout, id: &'static str, leftover: i64) -> GpuReader {
|
||||
let calls = l.data.join("calls");
|
||||
Arc::new(move || {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
@@ -747,21 +851,21 @@ mod tests {
|
||||
|
||||
/// A fake Strata that serves on `port` again (removes `gone`) and writes the engine's staging line.
|
||||
#[cfg(unix)]
|
||||
fn fake_placed(l: &Layout) -> std::path::PathBuf {
|
||||
pub(crate) fn fake_placed(l: &Layout) -> std::path::PathBuf {
|
||||
let log = l.strata_home().join("x.engine.log");
|
||||
fake_strata(l, &format!("rm -f '{}'; echo 'strata-q36: hybrid experts: (1.10 GiB staging); 7.1 s' >> '{}'; echo 'Strata is serving.'",
|
||||
fake_strata(l, &format!("rm -f '{}'; echo 'strata-q36: hybrid experts: 5 of 256 per layer in VRAM, the rest of a batch on 16 CPU threads (1.10 GiB staging); 7.1 s' >> '{}'; echo 'Strata is serving.'",
|
||||
l.data.join("gone").display(), log.display()))
|
||||
}
|
||||
|
||||
/// A fake server on a free port that answers only once a launch serves (removes `gone`).
|
||||
#[cfg(unix)]
|
||||
async fn server_after_launch(l: &Layout, mode: &'static str) -> u16 {
|
||||
pub(crate) async fn server_after_launch(l: &Layout, mode: &'static str) -> u16 {
|
||||
std::fs::create_dir_all(&l.data).unwrap();
|
||||
std::fs::write(l.data.join("gone"), "").unwrap();
|
||||
serve("qwen-test", Duration::from_millis(0), Some(l.data.join("gone")), Arc::new(Mutex::new(mode))).await
|
||||
}
|
||||
|
||||
fn launches(l: &Layout) -> Vec<String> {
|
||||
pub(crate) fn launches(l: &Layout) -> Vec<String> {
|
||||
std::fs::read_to_string(l.data.join("calls")).unwrap_or_default().lines()
|
||||
.filter(|c| c.starts_with("launch --family")).map(str::to_owned).collect()
|
||||
}
|
||||
@@ -841,6 +945,60 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn a_game_moves_the_llm_to_the_game_placement_and_back() {
|
||||
let l = layout("llm-stage");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(card(&l, "GPU-test", 0));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "normal", "gameRoomMib": null}));
|
||||
std::fs::remove_file(l.data.join("calls")).unwrap();
|
||||
assert_eq!(tokio::time::timeout(Duration::from_secs(30), llm.switch(&s, Stage::Gaming)).await.unwrap().unwrap(), Stage::Gaming);
|
||||
let g = launches(&l);
|
||||
// Measured once at the game's context and one request, then the 2 GiB minimum (8 GB does not
|
||||
// fit on 16 GB) with half the engine's 16 CPU threads.
|
||||
assert_eq!(g.len(), 2, "{g:?}");
|
||||
assert!(g[0].contains("--context 16384") && g[0].contains("--parallel 1") && g[0].contains("--set expert-vram=0"), "{}", g[0]);
|
||||
assert!(g[1].ends_with("--set expert-vram=2 --set prefill=2048 --set pool-workers=8"), "{}", g[1]);
|
||||
let room = 16303 - 1460 - 9874 - crate::placement::STT_MIB - crate::placement::MARGIN_MIB - 921;
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "gaming", "gameRoomMib": room}));
|
||||
assert_eq!(llm.status().state, LlmState::Ready);
|
||||
// Back to normal: the run's first reading of other programs, the normal placement again.
|
||||
std::fs::remove_file(l.data.join("calls")).unwrap();
|
||||
assert_eq!(llm.switch(&s, Stage::Normal).await.unwrap(), Stage::Normal);
|
||||
let n = launches(&l);
|
||||
assert_eq!(n.len(), 1, "{n:?}");
|
||||
assert!(n[0].contains("--context 32768") && n[0].ends_with("--set expert-vram=1.5 --set prefill=2048"), "{}", n[0]);
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "normal", "gameRoomMib": null}));
|
||||
// Pause: the LLM stops.
|
||||
assert_eq!(llm.switch(&s, Stage::Paused).await.unwrap(), Stage::Paused);
|
||||
assert_eq!(llm.status().state, LlmState::Off);
|
||||
assert_eq!(llm.stage_view()["stage"], json!("paused"));
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn a_game_with_no_room_beside_the_model_pauses_the_llm() {
|
||||
let l = layout("llm-stage-full");
|
||||
let port = server_after_launch(&l, "list").await;
|
||||
let calls = l.data.join("calls");
|
||||
// A 12 GB card: the model fits, but not with any room for a game.
|
||||
let llm = Llm::with_exe(l.clone(), fake_placed(&l)).with_gpu(Arc::new(move || {
|
||||
let c = std::fs::read_to_string(&calls).unwrap_or_default();
|
||||
let running = c.lines().last().is_some_and(|x| x.starts_with("launch --family"));
|
||||
Some(Gpu { id: "GPU-small".into(), used_mib: 1460 + if running { 9874 } else { 0 }, total_mib: 12288 })
|
||||
}));
|
||||
let s = parse_settings(&json!({"llm": {"port": port}})).unwrap();
|
||||
llm.start(&s).await.unwrap();
|
||||
assert_eq!(llm.switch(&s, Stage::Gaming).await.unwrap(), Stage::Paused);
|
||||
assert_eq!(llm.status().state, LlmState::Off);
|
||||
assert_eq!(llm.stage_view(), json!({"stage": "paused", "gameRoomMib": null}));
|
||||
let c = std::fs::read_to_string(l.data.join("calls")).unwrap();
|
||||
assert_eq!(c.lines().last(), Some("launch --stop"));
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn a_left_server_that_does_not_stop_means_no_experts_and_nothing_measured() {
|
||||
|
||||
@@ -156,6 +156,8 @@ struct Inner {
|
||||
events: mpsc::UnboundedSender<WorkerEvent>,
|
||||
ready: watch::Sender<Option<HelloOk>>,
|
||||
wake: Notify,
|
||||
/// The brain finished the answers it was giving when the app said the LLM switches.
|
||||
llm_idle: Notify,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
@@ -174,6 +176,7 @@ impl WorkerClient {
|
||||
events,
|
||||
ready: watch::channel(None).0,
|
||||
wake: Notify::new(),
|
||||
llm_idle: Notify::new(),
|
||||
});
|
||||
(Self { inner }, rx)
|
||||
}
|
||||
@@ -351,6 +354,22 @@ impl WorkerClient {
|
||||
self.send_json(json!({"type": "addressed", "id": id, "user": user.to_string(), "text": text}))
|
||||
}
|
||||
|
||||
/// Tell the brain the LLM's state (voice_worker.spec.md, "LLM switches"): `switching` (start no
|
||||
/// new LLM work, say `notice` once to the first spoken request that waits, report `llm_idle`),
|
||||
/// `ready` (answer what waited) or `paused` (answer nothing until ready).
|
||||
pub fn llm_state(&self, state: &str, notice: Option<&str>) -> bool {
|
||||
let mut m = json!({"type": "llm", "state": state});
|
||||
if let Some(n) = notice {
|
||||
m["notice"] = json!(n);
|
||||
}
|
||||
self.send_json(m)
|
||||
}
|
||||
|
||||
/// Wait until the brain reports `llm_idle` (also one that came before the wait), at most `t`.
|
||||
pub async fn wait_llm_idle(&self, t: Duration) -> bool {
|
||||
tokio::time::timeout(t, self.inner.llm_idle.notified()).await.is_ok()
|
||||
}
|
||||
|
||||
/// Tell the brain a call is being said (a partial transcript), so it gets ready to answer it.
|
||||
pub fn prepare(&self, user: u64, text: &str) -> bool {
|
||||
self.send_json(json!({"type": "prepare", "user": user.to_string(), "text": text}))
|
||||
@@ -524,6 +543,10 @@ fn on_text(inner: &Inner, t: &str) {
|
||||
"partial" => serde_json::from_value(v).ok().map(WorkerEvent::Partial),
|
||||
"final" => serde_json::from_value(v).ok().map(WorkerEvent::Final),
|
||||
"chat_action" => serde_json::from_value(v).ok().map(WorkerEvent::ChatAction),
|
||||
"llm_idle" => {
|
||||
inner.llm_idle.notify_one();
|
||||
return;
|
||||
}
|
||||
"reply_text" => {
|
||||
let id = v["id"].as_str().unwrap_or_default().to_owned();
|
||||
let failed = v["failed"].as_bool().unwrap_or(true);
|
||||
|
||||
@@ -26,7 +26,7 @@ fn a_launcher_game_turns_game_mode_on_until_it_ends() {
|
||||
eprintln!("skipped: set JARVIS_TEST_GAME_EXE");
|
||||
return;
|
||||
};
|
||||
let w = GameWatch::start(GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0 });
|
||||
let w = GameWatch::start(GameSettings { detect: true, games: vec![], not_games: vec![], other_room_gb: 1.0, reserve_gb: 8.0, revert_seconds: 30, pause_in_game: false });
|
||||
assert_eq!(w.view(), serde_json::Value::Null, "no game before it starts");
|
||||
let mut game = std::process::Command::new(&exe).spawn().expect("the game starts");
|
||||
let t0 = Instant::now();
|
||||
|
||||
@@ -414,3 +414,21 @@ async fn a_chat_reply_is_not_spoken_and_its_text_comes_back() {
|
||||
Got::Json(json!({"type": "reply_delivered", "id": "c2", "ok": false}))]);
|
||||
c.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_llm_switch_is_announced_and_the_client_waits_until_the_brain_is_idle() {
|
||||
let w = FakeWorker::start(0).await;
|
||||
let (c, _rx) = client(&w.url(), json!({}));
|
||||
c.start();
|
||||
c.wait_ready(Duration::from_secs(2)).await.unwrap();
|
||||
assert!(c.llm_state("switching", Some("잠깐만요")));
|
||||
until(|| w.got().iter().any(|g| *g == Got::Json(json!({"type": "llm", "state": "switching", "notice": "잠깐만요"})))).await;
|
||||
// Nothing idle yet: the wait gives up after its time.
|
||||
assert!(!c.wait_llm_idle(Duration::from_millis(100)).await);
|
||||
// The worker reports idle (before or while the app waits): the wait ends.
|
||||
w.send(Message::text(json!({"type": "llm_idle"}).to_string()));
|
||||
assert!(c.wait_llm_idle(Duration::from_secs(2)).await);
|
||||
assert!(c.llm_state("ready", None));
|
||||
until(|| w.got().iter().any(|g| *g == Got::Json(json!({"type": "llm", "state": "ready"})))).await;
|
||||
c.close();
|
||||
}
|
||||
|
||||
@@ -410,6 +410,14 @@ async function displayDriverCard() {
|
||||
|
||||
// ---- dashboard ----------------------------------------------------------------------------------
|
||||
|
||||
/** The performance stage for the dashboard; in a game also the graphics memory the game gets. */
|
||||
function perfText(p) {
|
||||
if (!p) return "평소";
|
||||
if (p.stage === "paused") return "일시정지 (게임 중)";
|
||||
if (p.stage === "gaming") return "게임 중" + (p.gameRoomMib != null ? ` · 게임에 ${(p.gameRoomMib / 1024).toFixed(1)} GB` : "");
|
||||
return "평소";
|
||||
}
|
||||
|
||||
const GAME_REASON = { fullscreen: () => "전체화면", borderless: () => "테두리 없는 창", registered: () => "직접 등록한 게임",
|
||||
launcher: (r) => `${r.launcher} 게임 폴더` };
|
||||
|
||||
@@ -471,7 +479,8 @@ function dashboardBody(d) {
|
||||
h("span", {}, `${g.usePercent}% · VRAM ${(g.vramUsedMb / 1024).toFixed(1)} / ${(g.vramTotalMb / 1024).toFixed(1)} GB`)]),
|
||||
h("span", { class: "k" }, "게임 모드"), h("span", { class: d.game ? "" : "muted" },
|
||||
d.game ? `켜짐 (${d.game.name}, ${GAME_REASON[d.game.reason.kind]?.(d.game.reason) ?? d.game.reason.kind})` : "꺼짐"),
|
||||
h("span", { class: "k" }, "성능 단계"), h("span", { class: "muted" }, d.performance ?? "평소")))),
|
||||
h("span", { class: "k" }, "성능 단계"), h("span", { class: d.performance && d.performance.stage !== "normal" ? "" : "muted" },
|
||||
perfText(d.performance))))),
|
||||
h("div", { class: "grid" },
|
||||
h("div", { class: "card" }, h("h2", {}, "실시간 자막"), h("div", { class: "captions" },
|
||||
(d.captions || []).length ? d.captions.map((c) => h("div", { class: "cap" + (c.final ? "" : " partial") }, h("b", {}, c.name), c.text))
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it is gated. This is the reference for optimising the app's main bottleneck (LLM latency). Keep it in sync with the code — see the note at the bottom.
|
||||
|
||||
**Transport (all contexts below).** `ollama_base_url` selects the API. A URL that ends in `/v1` (v2: Strata at `http://127.0.0.1:8080/v1`, which the app starts and writes with the model id Strata lists into the brain's `config/brain.json`, `app/src/strata.rs`; also llama.cpp or Ollama's own `/v1`) uses the OpenAI-compatible API: chat goes to `/chat/completions`, embeddings to `/embeddings`, and the warm-up is a one-token chat completion. Any other URL uses Ollama's `/api/chat`, `/api/generate` and `/api/embeddings`. The conversion lives in `src/jarvis/llm.py` only: callers keep sending Ollama-shaped messages and reading Ollama-shaped responses (`message.content`, `message.tool_calls` with dict arguments, `message.thinking` from `reasoning_content`). On the OpenAI path `num_ctx` and `keep_alive` are not sent (the server owns its context size and residency), `num_predict` becomes `max_tokens`, and the thinking switch becomes `chat_template_kwargs.enable_thinking`. The `num_ctx: 8192` limits listed below apply to the Ollama path. Ollama's own `/v1` ignores `num_ctx` and runs at its default context (2,048 tokens here), which cuts the system prompt; to test the OpenAI path on Ollama use a model tag created with a larger `num_ctx`. A server with no embedding model (Strata) answers `/embeddings` with an error and every embedding consumer fails open to its text path.
|
||||
**Transport (all contexts below).** `ollama_base_url` selects the API. A URL that ends in `/v1` (v2: Strata at `http://127.0.0.1:8080/v1`, which the app starts and writes with the model id Strata lists into the brain's `config/brain.json`, `app/src/strata.rs`; also llama.cpp or Ollama's own `/v1`) uses the OpenAI-compatible API: chat goes to `/chat/completions`, embeddings to `/embeddings`, and the warm-up is a one-token chat completion. Any other URL uses Ollama's `/api/chat`, `/api/generate` and `/api/embeddings`. The conversion lives in `src/jarvis/llm.py` only: callers keep sending Ollama-shaped messages and reading Ollama-shaped responses (`message.content`, `message.tool_calls` with dict arguments, `message.thinking` from `reasoning_content`). On the OpenAI path `num_ctx` and `keep_alive` are not sent (the server owns its context size and residency), `num_predict` becomes `max_tokens`, and the thinking switch becomes `chat_template_kwargs.enable_thinking`. The `num_ctx: 8192` limits listed below apply to the Ollama path. Ollama's own `/v1` ignores `num_ctx` and runs at its default context (2,048 tokens here), which cuts the system prompt; to test the OpenAI path on Ollama use a model tag created with a larger `num_ctx`. A server with no embedding model (Strata) answers `/embeddings` with an error and every embedding consumer fails open to its text path. In v2 the app also sets Strata's context per performance stage (`app/src/admin.spec.md`, "Performance stages"): the settings' `llm.context` (default 32,768 tokens) normally and 16,384 tokens with one request at a time while a game runs; while it switches the voice worker starts no LLM call (requests wait, at most 180 s, `voice_worker.spec.md` "LLM switches"), and while paused (a game with no room for the model) every brain call fails at once.
|
||||
|
||||
---
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -3,6 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import hmac
|
||||
import ipaddress
|
||||
import itertools
|
||||
@@ -38,6 +39,15 @@ def is_loopback(host: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
# How long a request waits for the LLM to come back from a switch (a restart takes 30-50 s; the
|
||||
# first game switch also measures once).
|
||||
LLM_WAIT_S = 180.0
|
||||
|
||||
|
||||
class _LlmPaused(Exception):
|
||||
"""The app paused the LLM (a game runs): nothing is answered until it is ready again."""
|
||||
|
||||
|
||||
class VoiceWorker:
|
||||
def __init__(self, token: str, transcriber: Transcriber | None, vad_factory: Callable[[], VadStream],
|
||||
tts_engine: TtsEngine | None = None, hello_timeout_s: float = 5.0, tick_s: float = 0.05,
|
||||
@@ -59,6 +69,16 @@ class VoiceWorker:
|
||||
# memory and delete confirmations ("the very next message") follow the order they asked.
|
||||
self.brain_lanes = {"voice": asyncio.Lock(), "chat": asyncio.Lock()}
|
||||
self.brain_people: dict[str, asyncio.Lock] = {}
|
||||
# The LLM's state from the app (voice_worker.spec.md, "LLM switches"): while it switches or
|
||||
# is paused no new brain work starts; the brain work running when a switch is announced
|
||||
# finishes first, then the app is told the brain is idle.
|
||||
self.llm_state = "ready"
|
||||
self.llm_ready = asyncio.Event()
|
||||
self.llm_ready.set()
|
||||
self.llm_notice: str | None = None
|
||||
self.llm_noticed = False
|
||||
self.llm_active = 0
|
||||
self.llm_idle_to = None # the session to tell once nothing runs
|
||||
self.chat_timeout_s = chat_timeout_s
|
||||
self.delivery_timeout_s = delivery_timeout_s # a chat answer the app never reports: not posted
|
||||
# chat_action ids are unique across connections and worker runs, so a late answer from an
|
||||
@@ -320,7 +340,7 @@ class Session:
|
||||
"addressed": self._on_addressed, "rank": self._on_rank,
|
||||
"chat_channels": self._on_chat_channels, "chat_action_result": self._on_chat_action_result,
|
||||
"memory_config": self._on_memory_config, "reply_delivered": self._on_reply_delivered,
|
||||
"prepare": self._on_prepare}.get(kind)
|
||||
"prepare": self._on_prepare, "llm": self._on_llm}.get(kind)
|
||||
if handler is None:
|
||||
self.error("bad_message", f"unknown type: {kind!r}")
|
||||
return
|
||||
@@ -519,6 +539,56 @@ class Session:
|
||||
waiting[user] = waiting.get(user, 0) + 1
|
||||
asyncio.get_running_loop().create_task(self._reply(rid, user, text, via, speak, early))
|
||||
|
||||
def _on_llm(self, msg: dict) -> None:
|
||||
state, notice = msg.get("state"), msg.get("notice")
|
||||
if state not in ("switching", "ready", "paused") or not (notice is None or isinstance(notice, str)):
|
||||
self.error("bad_message", "llm needs state switching|ready|paused and an optional string notice")
|
||||
return
|
||||
w = self.worker
|
||||
w.llm_state = state
|
||||
debug_log(f"llm {state}", "llm")
|
||||
if state == "ready":
|
||||
w.llm_notice, w.llm_idle_to = None, None
|
||||
w.llm_ready.set()
|
||||
return
|
||||
w.llm_ready.clear()
|
||||
if state == "switching":
|
||||
w.llm_notice, w.llm_noticed = notice or None, False
|
||||
w.llm_idle_to = self
|
||||
self._llm_check_idle()
|
||||
else:
|
||||
w.llm_ready.set() # paused: waiting requests stop waiting and answer nothing
|
||||
|
||||
def _llm_check_idle(self) -> None:
|
||||
w = self.worker
|
||||
if w.llm_state == "switching" and w.llm_active == 0 and w.llm_idle_to is not None:
|
||||
to, w.llm_idle_to = w.llm_idle_to, None
|
||||
to.send({"type": "llm_idle"})
|
||||
|
||||
@contextlib.asynccontextmanager
|
||||
async def _llm(self, notice_id: str | None = None):
|
||||
"""Brain work that uses the LLM: it waits while the LLM switches (the first spoken request
|
||||
that waits hears the notice once, under its own id), fails while paused, and counts as
|
||||
running so a switch waits for it."""
|
||||
w = self.worker
|
||||
while w.llm_state != "ready":
|
||||
if w.llm_state == "paused":
|
||||
raise _LlmPaused()
|
||||
if notice_id and w.llm_notice and not w.llm_noticed:
|
||||
w.llm_noticed = True
|
||||
# The newline ends it as a sentence, so it is spoken now, not joined to the answer.
|
||||
self._on_tts({"type": "tts", "id": notice_id, "text": w.llm_notice + "\n", "more": True})
|
||||
try:
|
||||
await asyncio.wait_for(w.llm_ready.wait(), LLM_WAIT_S)
|
||||
except asyncio.TimeoutError:
|
||||
break # go on: the call fails on its own if the LLM is still away
|
||||
w.llm_active += 1
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
w.llm_active -= 1
|
||||
self._llm_check_idle()
|
||||
|
||||
def _on_prepare(self, msg: dict) -> None:
|
||||
"""A call still being said: the brain gets ready for it. One prepare per user runs at a time;
|
||||
while it runs only the newest text waits, and one about the same as the last one prepared
|
||||
@@ -544,7 +614,8 @@ class Session:
|
||||
while text is not None and not self.closed:
|
||||
self.last_prepared[user] = text
|
||||
try:
|
||||
await asyncio.to_thread(self.worker.brain.prepare, user, text, bool(self.chat_channels))
|
||||
async with self._llm():
|
||||
await asyncio.to_thread(self.worker.brain.prepare, user, text, bool(self.chat_channels))
|
||||
except Exception as e: # the brain must never take the session down
|
||||
debug_log(f"brain prepare failed: {type(e).__name__}", "llm")
|
||||
newer, self.preparing[user] = self.preparing[user], None
|
||||
@@ -563,7 +634,8 @@ class Session:
|
||||
|
||||
async def judge():
|
||||
try:
|
||||
yes = await asyncio.to_thread(self.worker.brain.addressed, user, text, list(self.settings.hotwords))
|
||||
async with self._llm():
|
||||
yes = await asyncio.to_thread(self.worker.brain.addressed, user, text, list(self.settings.hotwords))
|
||||
except Exception as e: # the brain must never take the session down
|
||||
debug_log(f"brain addressed {qid} failed: {type(e).__name__}", "llm")
|
||||
yes = None
|
||||
@@ -585,7 +657,8 @@ class Session:
|
||||
|
||||
async def rank():
|
||||
try:
|
||||
order = await asyncio.to_thread(self.worker.brain.rank, cands)
|
||||
async with self._llm():
|
||||
order = await asyncio.to_thread(self.worker.brain.rank, cands)
|
||||
except Exception as e:
|
||||
debug_log(f"brain rank {qid} failed: {type(e).__name__}", "llm")
|
||||
order = None
|
||||
@@ -658,11 +731,14 @@ class Session:
|
||||
chat = ChatBridge(self, asyncio.get_running_loop(), turn, w.pending_deletes.setdefault(user, {})) \
|
||||
if self.chat_channels else None
|
||||
try:
|
||||
answer = await asyncio.to_thread(w.brain.reply, user, text, history, chat=chat)
|
||||
async with self._llm(rid if speak else None):
|
||||
answer = await asyncio.to_thread(w.brain.reply, user, text, history, chat=chat)
|
||||
except Exception as e: # the brain must never take the session down
|
||||
debug_log(f"brain reply {rid} failed: {type(e).__name__}", "llm")
|
||||
answer = None
|
||||
answer = (answer or "").strip()
|
||||
if not answer and rid in self.tts_requests:
|
||||
self._on_tts({"type": "tts", "id": rid, "text": "", "more": False}) # ends the notice's speech
|
||||
self.send({"type": "reply_text", "id": rid, "text": answer, "failed": not answer})
|
||||
# The answer reaches them when spoken (tts_done) or when the app reports the post
|
||||
# (reply_delivered); an unspoken voice answer has nothing more to wait for.
|
||||
|
||||
@@ -348,6 +348,29 @@ still deleted (below).
|
||||
- Each run is a `notice` line (category `voice_worker`): the target's name and how many new items, or
|
||||
a `Warning` with the error's type. Never a URL, words or collected text.
|
||||
|
||||
### LLM switches (M7)
|
||||
|
||||
The app restarts the LLM with another placement when a game starts or ends (app/src/admin.spec.md,
|
||||
"Performance stages"). It tells the worker first and waits for the brain to be idle:
|
||||
|
||||
```
|
||||
-> {"type": "llm", "state": "switching", "notice": "잠깐만요, 게임 모드로 바꾸는 중이에요"} // notice optional
|
||||
<- {"type": "llm_idle"} // nothing that uses the LLM runs any more
|
||||
-> {"type": "llm", "state": "ready"} // the LLM answers again
|
||||
-> {"type": "llm", "state": "paused"} // the LLM is stopped (a game runs)
|
||||
```
|
||||
|
||||
- Brain work that uses the LLM (`reply`, `prepare`, `addressed`, `rank`) counts as running from its
|
||||
start to its end. On `switching` new work waits; `llm_idle` goes to the connection that sent
|
||||
`switching` once nothing runs (at once when nothing does). On `ready` what waited goes on.
|
||||
- The first spoken `reply` that waits during one switch hears the notice under its own id (its
|
||||
answer follows in the same speech when ready); later ones wait without it. A notice ends with a
|
||||
newline, so it is spoken at once and never joined to the answer. A request waits at most 180 s,
|
||||
then goes on (its LLM call fails on its own if the LLM is still away).
|
||||
- On `paused` brain work answers nothing at once (`reply_text` `failed`, judgements failed) until
|
||||
`ready`; what waited for a switch stops waiting the same way.
|
||||
- Any other `state`, or a notice that is not a string, is `bad_message`.
|
||||
|
||||
### Errors
|
||||
|
||||
`{"type": "error", "code": "...", "message": "...", "id": "..."}`. Codes: `bad_message` (unknown type,
|
||||
|
||||
@@ -1130,3 +1130,96 @@ def test_worker_logs_name_their_service_so_the_app_puts_them_in_its_tab(store, m
|
||||
err = capfd.readouterr().err
|
||||
for cat, what in (("stt", "utterance end"), ("llm", "brain reply r1 failed"), ("tts", "tts sentence id=t1")):
|
||||
assert any(line.startswith(f"[{cat:^10}]") and what in line for line in err.splitlines()), (cat, err[-2000:])
|
||||
|
||||
|
||||
async def drain(ws, seconds):
|
||||
"""Every message that arrives within `seconds`."""
|
||||
out, end = [], asyncio.get_running_loop().time() + seconds
|
||||
while (left := end - asyncio.get_running_loop().time()) > 0:
|
||||
try:
|
||||
raw = await asyncio.wait_for(ws.recv(), left)
|
||||
except asyncio.TimeoutError:
|
||||
break
|
||||
out.append(raw if isinstance(raw, bytes) else json.loads(raw))
|
||||
return out
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_an_llm_switch_waits_for_the_answer_in_progress_then_holds_new_ones_with_one_notice(store):
|
||||
"""voice_worker.spec.md, "LLM switches": the app switches the LLM only once the brain is idle;
|
||||
requests meanwhile wait, the first spoken one hears the notice once, and all are answered once
|
||||
the LLM is ready."""
|
||||
import threading
|
||||
release = threading.Event()
|
||||
|
||||
def slow_first(user, text):
|
||||
if text == "first":
|
||||
release.wait(10)
|
||||
return f"answer for {text}"
|
||||
|
||||
tts = FakeTts()
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="voice_join", channel="vc1")
|
||||
await send(ws, type="reply", id="r1", user="100", text="first", via="voice", speak=True)
|
||||
await asyncio.sleep(0.1)
|
||||
await send(ws, type="llm", state="switching", notice="잠깐만요, 바꾸는 중이에요")
|
||||
# Not idle while the first answer is being written.
|
||||
seen = []
|
||||
try:
|
||||
seen += await asyncio.wait_for(recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "llm_idle"), 0.3)
|
||||
idle_early = True
|
||||
except asyncio.TimeoutError:
|
||||
idle_early = False
|
||||
release.set()
|
||||
seen += await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "llm_idle")
|
||||
# Two requests while switching: they wait; only the first spoken one hears the notice.
|
||||
await send(ws, type="reply", id="r2", user="200", text="second", via="voice", speak=True)
|
||||
await send(ws, type="reply", id="r3", user="300", text="third", via="voice", speak=True)
|
||||
seen += await drain(ws, 0.4)
|
||||
answered_while_switching = [m["id"] for m in seen if isinstance(m, dict) and m.get("type") == "reply_text"]
|
||||
await send(ws, type="llm", state="ready")
|
||||
seen += await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done" and m["id"] == "r3")
|
||||
return idle_early, answered_while_switching, seen
|
||||
|
||||
brain = FakeBrain(slow_first)
|
||||
idle_early, early, seen = asyncio.run(with_worker(body, store=store, brain=brain, tts=tts))
|
||||
assert not idle_early, "idle only after the answer in progress"
|
||||
assert "r2" not in early and "r3" not in early, "nothing new is answered while switching"
|
||||
assert [c[1] for c in brain.calls] == ["first", "second", "third"]
|
||||
assert tts.texts.count("잠깐만요, 바꾸는 중이에요") == 1, tts.texts
|
||||
done = {m["id"]: m for m in seen if isinstance(m, dict) and m.get("type") == "tts_done"}
|
||||
assert not done["r2"]["failed"] and not done["r3"]["failed"]
|
||||
# r2 heard the notice, then its answer, under its own id.
|
||||
assert tts.texts.index("잠깐만요, 바꾸는 중이에요") < tts.texts.index("answer for second")
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_while_paused_the_brain_answers_nothing(store):
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="llm", state="switching")
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "llm_idle")
|
||||
await send(ws, type="llm", state="paused")
|
||||
await send(ws, type="reply", id="r1", user="100", text="q", via="chat", speak=False)
|
||||
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
|
||||
brain = FakeBrain()
|
||||
seen = asyncio.run(with_worker(body, store=store, brain=brain))
|
||||
reply = [m for m in seen if isinstance(m, dict) and m.get("type") == "reply_text"][0]
|
||||
assert reply["failed"] and brain.calls == []
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_an_unknown_llm_state_is_refused(store):
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="llm", state="maybe")
|
||||
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "error")
|
||||
|
||||
seen = asyncio.run(with_worker(body, store=store, brain=FakeBrain()))
|
||||
assert [m for m in seen if isinstance(m, dict) and m.get("type") == "error"][0]["code"] == "bad_message"
|
||||
|
||||
Reference in New Issue
Block a user