feat(m0): measure the 1 GB room for other programs and a 95 % GPU cap

User decision (2026-11-10, room to watch a video next to Jarvis): the
normal placement leaves 1 GB of VRAM for programs opened later, and
Jarvis's GPU work is capped at 95 %.

- gamemode_tune's normal rows take OTHER_ROOM_MIB off the expert budget.
- gpucap.DutyCap pauses the Strata engine (strata.exe --serve under the
  kit) and the STT worker with their children together for 10 ms of
  every 200 ms (NtSuspendProcess), and always resumes them.
- Every row samples nvidia-smi utilization.gpu; capped rows have an
  uncapped twin (normal with and without the STT, game 2 GB with 2 s
  partials) so the run shows the cost and what the GPU really gets back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-10 23:52:57 +09:00
parent 3f0ed4343b
commit 0e50c558d1
5 changed files with 400 additions and 25 deletions

View File

@@ -68,7 +68,7 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
`run_m0_gamemode.bat`을 더블클릭하면 게임 중 자동 조절 단계 두 개(`gamemode`, `gamemode_stt`)만 돕니다. 먼저 LLM만 배치 5개(평소, 게임 중 후보 3개, 다시 평소)로 재고, 이어서 음성 인식을 함께 켠 배치 6개를 잽니다. 모델은 이미 받아 둔 것을 쓰므로 새로 받지 않고(음성 인식 모델을 아직 안 받았으면 약 1.6GB를 받습니다), 20~30분 걸립니다. 그동안 게임이나 무거운 프로그램은 꺼 두시고 PC에 손대지 않으시면 됩니다.
`run_m0_gamemode_tune.bat`을 더블클릭하면 `gamemode_tune` 단계만 돕니다. 앞 측정에서 정한 두 가지를 확인합니다. (1) 평소 배치에 음성 인식(STT) 자리를 남겼을 때의 속도와 첫 토큰 시간입니다. LLM을 전문가 층 없이 한 번 켜서 기본 VRAM을 재고, STT가 10명 동시 전사 때 더 쓰는 VRAM과 여유분을 뺀 나머지를 전문가 층에 줍니다. (2) 게임 중 배치(전문가 2GB)에서 LLM이 답하는 동안 부분 전사를 2초마다 하거나 멈추고 최종 전사만 바로 할 때, LLM 속도와 최종 전사가 기다린 시간입니다. 게임이 없을 때는 부분 전사를 줄이지 않으므로 평소 배치는 이 비교를 하지 않습니다. 게임은 켜지 않고 잽니다. 이미 받은 모델을 쓰고 10~15분 걸립니다. 그동안 게임이나 무거운 프로그램은 꺼 두시고 PC에 손대지 않으시면 됩니다.
`run_m0_gamemode_tune.bat`을 더블클릭하면 `gamemode_tune` 단계만 돕니다. 앞 측정에서 정한 세 가지를 확인합니다. (1) 평소 배치에 음성 인식(STT) 자리와 다른 프로그램용 여유 VRAM 1GB를 남겼을 때의 속도와 첫 토큰 시간입니다. LLM을 전문가 층 없이 한 번 켜서 기본 VRAM을 재고, STT가 10명 동시 전사 때 더 쓰는 VRAM, 여유분, 다른 프로그램용 1GB를 뺀 나머지를 전문가 층에 줍니다. (2) 게임 중 배치(전문가 2GB)에서 LLM이 답하는 동안 부분 전사를 2초마다 하거나 멈추고 최종 전사만 바로 할 때, LLM 속도와 최종 전사가 기다린 시간입니다. 게임이 없을 때는 부분 전사를 줄이지 않으므로 평소 배치는 이 비교를 하지 않습니다. (3) GPU 사용률 95% 제한입니다. LLM 엔진과 STT를 200ms마다 함께 10ms씩 멈추고, 제한이 없을 때와 비교해 속도와 실제 GPU 사용률을 잽니다. 이때 두 프로그램이 잠깐씩 멈추는 것은 정상입니다. 게임은 켜지 않고 잽니다. 이미 받은 모델을 쓰고 20~30분 걸립니다. 그동안 게임이나 무거운 프로그램은 꺼 두시고 PC에 손대지 않으시면 됩니다.
`run_m0_game_live.bat`을 더블클릭하면 실제 게임을 켠 상태로 잽니다(`gamemode_live`). 게임을 창 모드로 켜고 플레이 화면까지 들어간 뒤 Enter를 누르고 15초 안에 게임으로 돌아가면 됩니다. 게임 중 배치 4개(전문가 2GB·0GB, 음성 인식 GPU·CPU)를 차례로 재고 10~15분 걸리며, 끝나면 알림음이 울립니다. 게임 화면이나 게임 이름은 저장하지 않고 VRAM 사용량과 속도만 적습니다. 그래픽 메모리를 많이 쓰는 게임이면 배치가 켜지지 않을 수 있는데, 그것도 결과로 적습니다.

View File

@@ -49,7 +49,7 @@ STAGES = [
["model=fast", "only=together"], 0),
# The fixes for what gamemode_stt showed: room for the STT in the normal placement, and fewer
# STT partial passes while the LLM answers (normal and the 2 GB game placement).
("gamemode_tune", "게임 중 자동 조절: STT 자리 남기기와 부분 전사 간격", "base", "llm.py",
("gamemode_tune", "자동 조절: STT 자리와 VRAM 여유 1GB, 부분 전사 간격, GPU 95% 제한", "base", "llm.py",
["model=fast", "only=tune"], 0),
# With the user's game running. Needs someone at the PC, so only when asked for (--only).
("gamemode_live", "게임을 켠 상태로 재기 (직접 게임을 켜 주셔야 합니다)", "base", "llm.py",

201
scripts/m0/m0lib/gpucap.py Normal file
View File

@@ -0,0 +1,201 @@
"""The GPU share cap the app would put on Jarvis (user decision, 2026-11-10: 95 %).
Jarvis's GPU work runs in two processes: the Strata engine and the voice worker (here the STT
load worker). A GPU cannot be told to give one process only part of its time, so the cap pauses
both processes together for a short slice of every period (Windows NtSuspendProcess /
NtResumeProcess): 10 ms of every 200 ms is 95 %. Work already queued on the GPU still finishes
during the pause, so what the GPU really gets back is measured (``UtilSampler``), not assumed.
"""
from __future__ import annotations
import subprocess
import threading
import time
from pathlib import Path
from typing import Callable, Iterable
from m0lib.common import IS_WINDOWS, nvidia_smi, stats
PERIOD_S = 0.2
def gpu_util_pct() -> int | None:
"""nvidia-smi's utilization.gpu of GPU 0: the share of the last sample period in which any
kernel ran (what the app would read through NVML), or None."""
exe = nvidia_smi()
if not exe:
return None
try:
out = subprocess.run([exe, "--query-gpu=utilization.gpu", "--format=csv,noheader,nounits"],
capture_output=True, text=True, timeout=10).stdout.strip().splitlines()[0]
return int(out.strip())
except Exception:
return None
class UtilSampler:
"""Samples GPU utilisation in the background while a measurement runs."""
def __init__(self, interval: float = 0.5, read: Callable[[], int | None] | None = None):
self.interval = interval
self.read = read or gpu_util_pct
self.samples: list[int] = []
self._stop = threading.Event()
self._t: threading.Thread | None = None
def __enter__(self) -> "UtilSampler":
self._t = threading.Thread(target=self._loop, daemon=True)
self._t.start()
return self
def _loop(self) -> None:
while not self._stop.is_set():
v = self.read()
if v is not None:
self.samples.append(v)
self._stop.wait(self.interval)
def __exit__(self, *exc) -> None:
self._stop.set()
if self._t:
self._t.join(timeout=15)
def summary(self, cap: float | None = None) -> dict:
out = {"pct": stats(self.samples, 1)}
if cap is not None and self.samples:
out["over_cap_share"] = round(sum(1 for v in self.samples if v > cap * 100) / len(self.samples), 3)
return out
def process_table() -> list[dict]:
"""Every process with its id, parent, executable and command line (Windows only)."""
if not IS_WINDOWS:
return []
ps = ("Get-CimInstance Win32_Process | ForEach-Object { \"$($_.ProcessId)`t$($_.ParentProcessId)`t"
"$($_.ExecutablePath)`t$($_.CommandLine)\" }")
try:
out = subprocess.run(["powershell", "-NoProfile", "-Command", ps], capture_output=True, text=True,
encoding="utf-8", errors="replace", timeout=60).stdout
except Exception:
return []
return parse_process_table(out)
def parse_process_table(text: str) -> list[dict]:
rows = []
for line in text.splitlines():
parts = line.rstrip("\r").split("\t", 3)
if len(parts) < 4 or not parts[0].strip().isdigit():
continue
rows.append({"pid": int(parts[0]), "ppid": int(parts[1]) if parts[1].strip().isdigit() else None,
"exe": parts[2], "cmd": parts[3]})
return rows
def with_children(procs: list[dict], roots: Iterable[int]) -> list[int]:
"""The roots and everything they started (a venv python.exe on Windows is a launcher that
runs the real interpreter as its child)."""
out, todo = [], [r for r in roots if r is not None]
while todo:
pid = todo.pop()
if pid in out:
continue
out.append(pid)
todo += [p["pid"] for p in procs if p["ppid"] == pid]
return out
def engine_roots(procs: list[dict], home: Path) -> list[int]:
"""The Strata engine serving the model: its strata.exe --serve under our components folder
(not the strata terminal that started it)."""
where = str(home / "components").lower()
return [p["pid"] for p in procs
if p["exe"].lower().startswith(where) and p["exe"].lower().endswith("strata.exe")
and "--serve" in p["cmd"]]
_API: dict = {}
def _nt(name: str, pid: int) -> bool:
import ctypes
from ctypes import wintypes
if not _API:
k32 = ctypes.WinDLL("kernel32", use_last_error=True)
ntdll = ctypes.WinDLL("ntdll")
k32.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]
k32.OpenProcess.restype = ctypes.c_void_p
k32.CloseHandle.argtypes = [ctypes.c_void_p]
for fn in (ntdll.NtSuspendProcess, ntdll.NtResumeProcess):
fn.argtypes = [ctypes.c_void_p]
fn.restype = ctypes.c_long
_API.update(k32=k32, ntdll=ntdll)
k32 = _API["k32"]
h = k32.OpenProcess(0x0800, False, pid) # PROCESS_SUSPEND_RESUME
if not h:
return False
try:
return getattr(_API["ntdll"], name)(h) == 0
finally:
k32.CloseHandle(h)
def suspend_pid(pid: int) -> bool:
return _nt("NtSuspendProcess", pid)
def resume_pid(pid: int) -> bool:
return _nt("NtResumeProcess", pid)
class DutyCap:
"""Pause ``pids`` together for (1 - cap) of every ``period`` s while the block runs; every
paused process is resumed before the block ends, also on an error."""
def __init__(self, pids: list[int], cap: float, period: float | None = None,
suspend: Callable[[int], bool] | None = None, resume: Callable[[int], bool] | None = None):
self.pids, self.cap, self.period = list(pids), cap, period or PERIOD_S
self.suspend, self.resume = suspend or suspend_pid, resume or resume_pid
self.pause = self.period * (1 - cap)
self.paused_s = 0.0
self.cycles = 0
self.failed: set[int] = set()
self._stop = threading.Event()
self._t: threading.Thread | None = None
self._t0 = 0.0
self._t1 = 0.0
def __enter__(self) -> "DutyCap":
self._t0 = time.perf_counter()
if self.pids and self.pause > 0:
self._t = threading.Thread(target=self._loop, daemon=True)
self._t.start()
return self
def _loop(self) -> None:
while not self._stop.wait(self.period - self.pause):
held = [p for p in self.pids if self.suspend(p)]
self.failed.update(p for p in self.pids if p not in held)
s0 = time.perf_counter()
try:
time.sleep(self.pause)
finally:
for p in held:
if not self.resume(p):
self.failed.add(p)
self.paused_s += time.perf_counter() - s0
self.cycles += 1
def __exit__(self, *exc) -> None:
self._stop.set()
if self._t:
self._t.join(timeout=10)
self._t1 = time.perf_counter()
def summary(self) -> dict:
span = (self._t1 or time.perf_counter()) - self._t0
return {"cap": self.cap, "period_s": self.period, "pause_s": round(self.pause, 4),
"processes": len(self.pids), "cycles": self.cycles,
"paused_share": round(self.paused_s / span, 3) if span > 0 else None,
"failed_processes": len(self.failed)}

View File

@@ -10,8 +10,9 @@ Options (--opt key=value):
only=together the same with the STT loaded next to it and transcribing ten
speakers at once, on the GPU and on the CPU (needs stt_python=)
only=live game placements and the STT while the user runs a real game
only=tune the normal placement with room left for the STT, and in the game
placement the STT doing fewer partial passes while the LLM answers
only=tune the normal placement with room left for the STT and 1 GB for other
programs, the game placement with the STT doing fewer partial passes
while the LLM answers, and both with Jarvis capped at 95 % of the GPU
stt_python=PATH the Python of the stt venv (m0.py passes it)
"""
@@ -34,6 +35,7 @@ from m0lib.common import (IS_WINDOWS, GpuSampler, StageResult, ask, gpu_memory_m
from m0lib.llm_bench import (bench_single, quick_bench, test_decode, test_images, test_parallel, # noqa: E402
test_tools, test_vram_shrink, warmup)
from m0lib.tool_fix import test_tools_fixed, tools_passed # noqa: E402
from m0lib import gpucap # noqa: E402
from m0lib.llm_client import ChatClient, http_json # noqa: E402
PRESETS = {
@@ -111,18 +113,30 @@ LIVE = [lv for lv in TOGETHER if lv["key"].startswith("game_")]
# whole run. Without a game the STT keeps every partial (user decision), so only the game
# placement is paced; its back-to-back row is the baseline measured in the same run.
# The 2 GB expert placement is the game minimum on a 16 GB card (faster than 0 GB in a real game).
# Then (user decision, 2026-11-10, for watching a video next to Jarvis):
# - the normal placement also leaves OTHER_ROOM_MIB free for programs opened later ("other_room_mib");
# - Jarvis's GPU work (the engine and the STT worker together) is paused for 5 % of every 200 ms
# ("gpu_cap", gpucap.DutyCap). The rows with and without it measure what it costs and how much
# GPU time it really gives back (queued work still runs during the pause).
STT_ROOM_MARGIN_MIB = 768 # what Strata's qwen36 engine itself keeps free when it places experts
STT_GROWTH_FALLBACK_MIB = 1440 # M0 stt on the game PC: ten speakers at once over the loaded model
OTHER_ROOM_MIB = 1024 # the app's default room for other programs in the normal placement
GPU_CAP = 0.95
_PARTIAL_2S = {"partial_every": 2.0, "final_every": 0.5}
_FINALS_ONLY = {"partial_every": None, "final_every": 0.5}
_ROOM = {"parallel": 1, "vision": "gpu", "sets": ["prefill=2048"], "stt": "cuda", "stt_room": True}
_ROOM = {"parallel": 1, "vision": "gpu", "sets": ["prefill=2048"], "stt_room": True,
"other_room_mib": OTHER_ROOM_MIB}
_GAME2 = {**_GAME, "context": 16384, "sets": _EXP2, "stt": "cuda"}
TUNED = [
{"key": "normal_stt_room_stt_gpu", **_ROOM},
{"key": "game_experts_2g_16k_stt_gpu", **_GAME, "context": 16384, "sets": _EXP2, "stt": "cuda"},
{"key": "game_experts_2g_16k_partial_2s", **_GAME, "context": 16384, "sets": _EXP2, "stt": "cuda",
"stt_pace": _PARTIAL_2S},
{"key": "game_experts_2g_16k_finals_only", **_GAME, "context": 16384, "sets": _EXP2, "stt": "cuda",
"stt_pace": _FINALS_ONLY},
# The room is worked out in the first row, with the STT loaded; the rows after it reuse it.
{"key": "normal_room_stt_gpu", **_ROOM, "stt": "cuda"},
{"key": "normal_room_stt_gpu_cap95", **_ROOM, "stt": "cuda", "gpu_cap": GPU_CAP},
{"key": "normal_room_llm_only", **_ROOM},
{"key": "normal_room_llm_only_cap95", **_ROOM, "gpu_cap": GPU_CAP},
{"key": "game_experts_2g_16k_stt_gpu", **_GAME2},
{"key": "game_experts_2g_16k_partial_2s", **_GAME2, "stt_pace": _PARTIAL_2S},
{"key": "game_experts_2g_16k_partial_2s_cap95", **_GAME2, "stt_pace": _PARTIAL_2S, "gpu_cap": GPU_CAP},
{"key": "game_experts_2g_16k_finals_only", **_GAME2, "stt_pace": _FINALS_ONLY},
]
LEVEL_SETS = {"levels": LEVELS, "together": TOGETHER, "live": LIVE, "tune": TUNED}
INSTALL_URL = "https://github.com/Niko1221/Strata/releases/latest/download/install.ps1"
@@ -298,6 +312,10 @@ class SttLoad:
self.cmd += ["--opt", f"{key}={'none' if v is None else v}"]
self.proc: subprocess.Popen | None = None
@property
def pid(self) -> int | None:
return self.proc.pid if self.proc else None
def _read(self) -> dict:
try:
return read_json(self.out)
@@ -463,8 +481,12 @@ def measure_levels(r: StageResult, s: Strata, preset: dict, context: int, levels
continue
sets = list(lv["sets"])
if lv.get("stt_room"):
where = (row["context"], lv["parallel"], lv["vision"])
where = (row["context"], lv["parallel"], lv["vision"], lv.get("other_room_mib", 0))
if where not in rooms:
if not stt:
# The probe needs the STT model in VRAM, as the app has it.
r.error(f"level {lv['key']}", "no STT room was measured with the STT loaded")
continue
rooms[where] = stt_room(r, s, preset, row, lv, base)
row["stt_room"] = rooms[where]
sets.append(f"expert-vram={row['stt_room']['expert_vram_gib']}")
@@ -483,7 +505,18 @@ def measure_levels(r: StageResult, s: Strata, preset: dict, context: int, levels
c.chat([{"role": "user", "content": "안녕"}], max_tokens=8) # loading is not answer time
if stt:
stt.go()
row.update(quick_bench(c))
cap = lv.get("gpu_cap")
pids = cap_pids(s, stt) if cap else []
if cap and not pids:
r.error(f"level {lv['key']} gpu cap", "found no engine process to pause")
with gpucap.UtilSampler() as util, gpucap.DutyCap(pids, cap or 1.0) as duty:
row.update(quick_bench(c))
row["gpu_util"] = util.summary(cap)
if cap:
row["gpu_cap"] = duty.summary()
if row["gpu_cap"]["failed_processes"]:
r.error(f"level {lv['key']} gpu cap",
f"{row['gpu_cap']['failed_processes']} process(es) could not be paused")
for err in row["errors"]:
r.error(f"level {lv['key']}", err)
except Exception as e:
@@ -540,10 +573,11 @@ def staging_gib(lines: list[str], default: float = 1.10) -> float:
def expert_budget_gib(total_mib: int, used_mib: int, probe_model_mib: int, staging: float, growth_mib: int,
margin_mib: int = STT_ROOM_MARGIN_MIB) -> float:
"""--expert-vram (GiB, rounded down to 0.1) that leaves the STT's growth and the margin free.
The budget first holds the staging, which the start without experts already has."""
spare = total_mib - used_mib - probe_model_mib - growth_mib - margin_mib
margin_mib: int = STT_ROOM_MARGIN_MIB, other_mib: int = 0) -> float:
"""--expert-vram (GiB, rounded down to 0.1) that leaves the STT's growth, the margin and the room
for other programs free. The budget first holds the staging, which the start without experts
already has."""
spare = total_mib - used_mib - probe_model_mib - growth_mib - margin_mib - other_mib
if spare <= 0:
return 0.0
return math.floor((staging * 1024 + spare) / 1024 * 10) / 10
@@ -553,7 +587,7 @@ def stt_room(r: StageResult, s: Strata, preset: dict, row: dict, lv: dict, base:
"""Start the placement once without experts (the STT model already loaded), see what the LLM
holds, stop it, and work out the expert budget that leaves the STT's ten-speaker growth free."""
room: dict = {"stt_growth_mib": stt_growth_mib(s.log.parent.parent), "margin_mib": STT_ROOM_MARGIN_MIB,
"expert_vram_gib": 0.0}
"other_room_mib": lv.get("other_room_mib", 0), "expert_vram_gib": 0.0}
say(" STT 자리 계산: 전문가 층 없이 한 번 켜서 LLM 기본 VRAM을 잽니다")
before = gpu_memory_mib()
launch = s.launch(preset, row["context"], parallel=lv["parallel"], vision=lv["vision"],
@@ -571,7 +605,8 @@ def stt_room(r: StageResult, s: Strata, preset: dict, row: dict, lv: dict, base:
room.update(probe_model_mib=after[0] - before[0], used_before_mib=before[0], total_mib=before[1],
staging_gib=staging_gib(lines))
room["expert_vram_gib"] = expert_budget_gib(before[1], before[0], room["probe_model_mib"],
room["staging_gib"], room["stt_growth_mib"])
room["staging_gib"], room["stt_growth_mib"],
other_mib=room["other_room_mib"])
if room["expert_vram_gib"] == 0:
r.note(f"{lv['key']}: no room for experts next to the STT; started with --expert-vram 0")
else:
@@ -580,6 +615,16 @@ def stt_room(r: StageResult, s: Strata, preset: dict, row: dict, lv: dict, base:
return room
def cap_pids(s: Strata, stt: SttLoad | None) -> list[int]:
"""The processes the GPU cap pauses together: the engine and the STT worker, with their children.
None when the engine is not found (pausing only the STT would not be the cap)."""
procs = gpucap.process_table()
roots = gpucap.engine_roots(procs, s.home)
if not roots:
return []
return gpucap.with_children(procs, roots + ([stt.pid] if stt and stt.pid else []))
def ask_for_a_game() -> bool:
"""The live stage needs someone to start a game first. False when nobody is there or they say no."""
if not interactive():