feat(m0): measure the normal placement with room for the STT and paced STT partials
The gamemode_stt run showed Strata's automatic placement filling the card next to the STT (15.9 of 16.3 GB, first token 1.6 -> 7.5 s) and the STT's back-to-back partial passes cutting LLM output to a third. The new gamemode_tune stage measures both fixes the user chose: - stt_room: one start without experts (STT loaded) gives the LLM's base VRAM; --expert-vram gets what is left after the STT's ten-speaker growth (from this PC's stt stage) and a 768 MiB margin. - stt_pace: a ten-speaker partial pass every 2 s, or none, with finals every 0.5 s and finals first; records final wait and STT GPU share. run_m0_gamemode_tune.bat runs it unattended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -37,6 +37,7 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
|
||||
| PC 제어 도구 | agent-desktop으로 계산기 버튼을 눌러 12+3=15가 나오는지, 테스트용 브라우저 창(새 임시 프로필)의 입력칸에 글자를 넣고 다시 읽히는지, 디스코드·탐색기 화면을 읽는 시간, 계속 켜 두는 방식(MCP)과 매번 실행하는 방식의 명령 한 번 시간 |
|
||||
| 게임 중 자동 조절 | 이미 받은 Qwen3.6-35B-A3B를 평소 배치(동시 1개)와 게임 중 배치 3개(전문가 층 2GB 맥락 32K, 2GB 맥락 16K, 0GB 맥락 16K)로 차례로 다시 시작해서, 배치마다 전환 시간, 남는 VRAM, 출력 속도, 음성 첫 문장, 맥락 8천 토큰 첫 토큰 시간을 잽니다. 마지막에 평소 배치로 한 번 더 켜서 되돌리는 시간도 잽니다 |
|
||||
| LLM과 음성 인식 함께 | 음성 인식(STT)을 먼저 켜 두고 LLM을 평소 배치(동시 1개, 2개)와 게임 중 배치(전문가 2GB·0GB, 맥락 16K)로 켭니다. LLM을 재는 동안 음성 인식은 10명 동시 전사를 계속 돌립니다. 게임 중 배치는 음성 인식을 GPU에 둔 경우와 CPU에 둔 경우를 둘 다 잽니다. 배치마다 남는 VRAM, LLM 속도, 10명 동시 전사 시간, 한 사람 전사 시간을 적습니다 |
|
||||
| STT 자리와 부분 전사 간격 | 평소 배치를 STT 자리를 남긴 전문가 층 몫으로 켜고, STT를 쉬지 않고 돌릴 때, 부분 전사를 2초마다 할 때, 부분 전사 없이 최종 전사만 할 때를 잽니다. 게임 중 배치(전문가 2GB)도 뒤의 두 경우를 잽니다. 배치마다 남는 VRAM, LLM 속도, 최종 전사 시간과 기다린 시간, STT가 GPU를 쓴 비율을 적습니다 |
|
||||
|
||||
## 개인정보
|
||||
|
||||
@@ -55,7 +56,7 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
|
||||
.\run_m0.ps1 -Only stt,tts_melo -Redo # 특정 단계만 다시
|
||||
```
|
||||
|
||||
단계 이름: `sysinfo`, `winchecks`, `vdisplay`, `tts_melo`, `tts_cosy2`, `tts_cosy3`, `stt`, `llm_fast`, `llm_recommended`, `tools_fast`, `tools_recommended`, `desktop`, `gamemode`, `gamemode_stt`, `gamemode_live`(이 단계는 `-Only`로 고를 때만 돕니다)
|
||||
단계 이름: `sysinfo`, `winchecks`, `vdisplay`, `tts_melo`, `tts_cosy2`, `tts_cosy3`, `stt`, `llm_fast`, `llm_recommended`, `tools_fast`, `tools_recommended`, `desktop`, `gamemode`, `gamemode_stt`, `gamemode_tune`, `gamemode_live`(이 단계는 `-Only`로 고를 때만 돕니다)
|
||||
|
||||
`tools_fast`, `tools_recommended`는 도구 호출만 다시 잽니다. 처음과 같은 방식(고치기 전)과, 자비스의 수정(지시문 규칙, 계획 단계, 건너뛴 도구 직접 실행)을 넣은 방식을 같은 모델로 비교합니다. 이미 받은 모델을 쓰므로 모델 하나에 몇 분이면 끝납니다.
|
||||
|
||||
@@ -67,6 +68,8 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
|
||||
|
||||
`run_m0_gamemode.bat`을 더블클릭하면 게임 중 자동 조절 단계 두 개(`gamemode`, `gamemode_stt`)만 돕니다. 먼저 LLM만 배치 5개(평소, 게임 중 후보 3개, 다시 평소)로 재고, 이어서 음성 인식을 함께 켠 배치 6개를 잽니다. 모델은 이미 받아 둔 것을 쓰므로 새로 받지 않고(음성 인식 모델을 아직 안 받았으면 약 1.6GB를 받습니다), 20~30분 걸립니다. 그동안 게임이나 무거운 프로그램은 꺼 두시고 PC에 손대지 않으시면 됩니다.
|
||||
|
||||
`run_m0_gamemode_tune.bat`을 더블클릭하면 `gamemode_tune` 단계만 돕니다. 앞 측정에서 정한 두 가지를 확인합니다. (1) 평소 배치에 음성 인식(STT) 자리를 남겼을 때의 속도와 첫 토큰 시간입니다. LLM을 전문가 층 없이 한 번 켜서 기본 VRAM을 재고, STT가 10명 동시 전사 때 더 쓰는 VRAM과 여유분을 뺀 나머지를 전문가 층에 줍니다. (2) LLM이 답하는 동안 부분 전사를 2초마다 하거나 멈추고 최종 전사만 바로 할 때, LLM 속도와 최종 전사가 기다린 시간입니다. 평소 배치와 게임 중 배치(전문가 2GB)에서 잽니다. 이미 받은 모델을 쓰고 15~20분 걸립니다. 그동안 게임이나 무거운 프로그램은 꺼 두시고 PC에 손대지 않으시면 됩니다.
|
||||
|
||||
`run_m0_game_live.bat`을 더블클릭하면 실제 게임을 켠 상태로 잽니다(`gamemode_live`). 게임을 창 모드로 켜고 플레이 화면까지 들어간 뒤 Enter를 누르고 15초 안에 게임으로 돌아가면 됩니다. 게임 중 배치 4개(전문가 2GB·0GB, 음성 인식 GPU·CPU)를 차례로 재고 10~15분 걸리며, 끝나면 알림음이 울립니다. 게임 화면이나 게임 이름은 저장하지 않고 VRAM 사용량과 속도만 적습니다. 그래픽 메모리를 많이 쓰는 게임이면 배치가 켜지지 않을 수 있는데, 그것도 결과로 적습니다.
|
||||
|
||||
다 끝나고 필요 없으면 작업 폴더(`\JarvisM0`)를 통째로 지우면 됩니다. Python, 모델, Strata(엔진과 설정 포함)는 모두 그 안에 설치하도록 지정해 두었습니다.
|
||||
|
||||
@@ -47,6 +47,10 @@ STAGES = [
|
||||
# The same placements with the STT loaded next to the LLM (on the GPU and on the CPU).
|
||||
("gamemode_stt", "게임 중 자동 조절: LLM과 음성 인식을 함께 켰을 때", "base", "llm.py",
|
||||
["model=fast", "only=together"], 0),
|
||||
# The fixes for what gamemode_stt showed: room for the STT in the normal placement, and fewer
|
||||
# STT partial passes while the LLM answers (normal and the 2 GB game placement).
|
||||
("gamemode_tune", "게임 중 자동 조절: STT 자리 남기기와 부분 전사 간격", "base", "llm.py",
|
||||
["model=fast", "only=tune"], 0),
|
||||
# With the user's game running. Needs someone at the PC, so only when asked for (--only).
|
||||
("gamemode_live", "게임을 켠 상태로 재기 (직접 게임을 켜 주셔야 합니다)", "base", "llm.py",
|
||||
["model=fast", "only=live"], 0),
|
||||
@@ -54,7 +58,7 @@ STAGES = [
|
||||
# Stages left out of a full run: they need the user there after the unattended part.
|
||||
ON_DEMAND = {"gamemode_live"}
|
||||
# Stages that start the STT worker next to the LLM and need the stt venv's Python.
|
||||
NEEDS_STT = {"gamemode_stt", "gamemode_live"}
|
||||
NEEDS_STT = {"gamemode_stt", "gamemode_tune", "gamemode_live"}
|
||||
|
||||
|
||||
def free_gb(path: Path) -> float:
|
||||
|
||||
@@ -10,12 +10,15 @@ Options (--opt key=value):
|
||||
only=together the same with the STT loaded next to it and transcribing ten
|
||||
speakers at once, on the GPU and on the CPU (needs stt_python=)
|
||||
only=live game placements and the STT while the user runs a real game
|
||||
only=tune the normal placement with room left for the STT, and the STT
|
||||
doing fewer partial passes while the LLM answers
|
||||
stt_python=PATH the Python of the stt venv (m0.py passes it)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
@@ -96,7 +99,31 @@ TOGETHER = [
|
||||
]
|
||||
# With a real game running: only game placements (the normal one would take the game's VRAM).
|
||||
LIVE = [lv for lv in TOGETHER if lv["key"].startswith("game_")]
|
||||
LEVEL_SETS = {"levels": LEVELS, "together": TOGETHER, "live": LIVE}
|
||||
# The fixes for what gamemode_stt showed (user decision, 2026-11-10):
|
||||
# - Strata's automatic placement filled the card with experts and left the STT no room (15.9 of
|
||||
# 16.3 GB with the STT busy; the first token after a long context went from 1.6 to 7.5 s). The
|
||||
# qwen36 engine ignores --vram-reserve-mib, so "stt_room" sets --expert-vram from what this PC
|
||||
# measures: one start without experts (with the STT loaded) gives what the LLM needs before any
|
||||
# expert; the experts get what is left after the STT's ten-speaker growth and a margin.
|
||||
# - The STT's partial passes took the GPU from the LLM (output 1/2.5 to 1/3). "stt_pace" is what
|
||||
# the app would do while the LLM answers: a ten-speaker partial pass every 2 s, or none, and the
|
||||
# finals at once. The LLM is measured the whole time, so the pace holds for the whole run.
|
||||
# The 2 GB expert placement is the game minimum on a 16 GB card (faster than 0 GB in a real game).
|
||||
STT_ROOM_MARGIN_MIB = 768 # what Strata's qwen36 engine itself keeps free when it places experts
|
||||
STT_GROWTH_FALLBACK_MIB = 1440 # M0 stt on the game PC: ten speakers at once over the loaded model
|
||||
_PARTIAL_2S = {"partial_every": 2.0, "final_every": 0.5}
|
||||
_FINALS_ONLY = {"partial_every": None, "final_every": 0.5}
|
||||
_ROOM = {"parallel": 1, "vision": "gpu", "sets": ["prefill=2048"], "stt": "cuda", "stt_room": True}
|
||||
TUNED = [
|
||||
{"key": "normal_stt_room_stt_gpu", **_ROOM},
|
||||
{"key": "normal_stt_room_partial_2s", **_ROOM, "stt_pace": _PARTIAL_2S},
|
||||
{"key": "normal_stt_room_finals_only", **_ROOM, "stt_pace": _FINALS_ONLY},
|
||||
{"key": "game_experts_2g_16k_partial_2s", **_GAME, "context": 16384, "sets": _EXP2, "stt": "cuda",
|
||||
"stt_pace": _PARTIAL_2S},
|
||||
{"key": "game_experts_2g_16k_finals_only", **_GAME, "context": 16384, "sets": _EXP2, "stt": "cuda",
|
||||
"stt_pace": _FINALS_ONLY},
|
||||
]
|
||||
LEVEL_SETS = {"levels": LEVELS, "together": TOGETHER, "live": LIVE, "tune": TUNED}
|
||||
INSTALL_URL = "https://github.com/Niko1221/Strata/releases/latest/download/install.ps1"
|
||||
|
||||
|
||||
@@ -255,7 +282,7 @@ class Strata:
|
||||
class SttLoad:
|
||||
"""The STT worker (stt.py mode=load in the stt venv) as a separate process next to the LLM."""
|
||||
|
||||
def __init__(self, python: str, work: Path, device: str, log: Path):
|
||||
def __init__(self, python: str, work: Path, device: str, log: Path, pace: dict | None = None):
|
||||
d = work / "cache" / f"stt_load_{device}"
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
d.mkdir(parents=True)
|
||||
@@ -264,6 +291,10 @@ class SttLoad:
|
||||
self.cmd = [python, str(Path(__file__).resolve().parent / "stt.py"), "--out", str(self.out),
|
||||
"--work", str(work), "--opt", "mode=load", "--opt", f"device={device}",
|
||||
"--opt", f"go={self.go_file}", "--opt", f"stop={self.stop_file}"]
|
||||
if pace is not None:
|
||||
for key in ("partial_every", "final_every"):
|
||||
v = pace.get(key)
|
||||
self.cmd += ["--opt", f"{key}={'none' if v is None else v}"]
|
||||
self.proc: subprocess.Popen | None = None
|
||||
|
||||
def _read(self) -> dict:
|
||||
@@ -307,15 +338,17 @@ class SttLoad:
|
||||
self._logf.close()
|
||||
|
||||
|
||||
def stt_gaps(busy: dict) -> list[str]:
|
||||
def stt_gaps(busy: dict, pace: dict | None = None) -> list[str]:
|
||||
"""Why an STT load next to the LLM did not measure what it should: its errors, a worker that
|
||||
did not finish (timed out, died, never started), or no ten-speaker or single-clip timings."""
|
||||
did not finish (timed out, died, never started), or no ten-speaker or single-clip timings
|
||||
(no ten-speaker ones are expected when the pace turns the partials off)."""
|
||||
gaps = list(busy.get("errors") or [])
|
||||
if busy.get("status") != "done":
|
||||
if not gaps:
|
||||
gaps.append(f"stt worker ended as {busy.get('status') or 'nothing written'}")
|
||||
return gaps
|
||||
for key in ("batch10_s", "single_s"):
|
||||
partials = pace is None or pace.get("partial_every") is not None
|
||||
for key in (("batch10_s", "single_s") if partials else ("single_s",)):
|
||||
if not (busy.get(key) or {}).get("n"):
|
||||
gaps.append(f"stt worker measured no {key}")
|
||||
return gaps
|
||||
@@ -406,6 +439,7 @@ def measure_levels(r: StageResult, s: Strata, preset: dict, context: int, levels
|
||||
base = f"http://127.0.0.1:{PORT}/v1"
|
||||
rows: list[dict] = []
|
||||
r["levels"] = rows
|
||||
rooms: dict = {} # one probe per placement that leaves the STT room
|
||||
for lv in levels:
|
||||
row: dict = {"key": lv["key"], "parallel": lv["parallel"], "vision": lv["vision"], "sets": lv["sets"],
|
||||
"context": lv.get("context", context), "kv": lv.get("kv"), "mtp": lv.get("mtp")}
|
||||
@@ -419,14 +453,24 @@ def measure_levels(r: StageResult, s: Strata, preset: dict, context: int, levels
|
||||
with GpuSampler(interval=1) as g:
|
||||
try:
|
||||
if lv.get("stt"):
|
||||
stt = SttLoad(stt_python, s.log.parent.parent, lv["stt"], s.log.with_name("stt_load.log"))
|
||||
row["stt"] = {"device": lv["stt"], "load": stt.start()}
|
||||
stt = SttLoad(stt_python, s.log.parent.parent, lv["stt"], s.log.with_name("stt_load.log"),
|
||||
pace=lv.get("stt_pace"))
|
||||
row["stt"] = {"device": lv["stt"], "pace": lv.get("stt_pace"), "load": stt.start()}
|
||||
if row["stt"]["load"].get("status") != "loaded":
|
||||
r.error(f"level {lv['key']}", "; ".join(row["stt"]["load"].get("errors") or [])
|
||||
or "the STT did not load")
|
||||
continue
|
||||
sets = list(lv["sets"])
|
||||
if lv.get("stt_room"):
|
||||
where = (row["context"], lv["parallel"], lv["vision"])
|
||||
if where not in rooms:
|
||||
rooms[where] = stt_room(r, s, preset, row, lv, base)
|
||||
row["stt_room"] = rooms[where]
|
||||
sets.append(f"expert-vram={row['stt_room']['expert_vram_gib']}")
|
||||
row["sets"] = sets
|
||||
t0 = time.perf_counter() # the probe is not part of the switch
|
||||
row["launch"] = s.launch(preset, row["context"], parallel=lv["parallel"], vision=lv["vision"],
|
||||
extra_sets=lv["sets"], kv=lv.get("kv"), mtp=lv.get("mtp"), timeout=3600)
|
||||
extra_sets=sets, kv=lv.get("kv"), mtp=lv.get("mtp"), timeout=3600)
|
||||
ready = row["launch"].get("ready")
|
||||
row["switch_s"] = round(time.perf_counter() - t0, 1) if ready else None
|
||||
row["engine"] = s.engine_lines()
|
||||
@@ -446,8 +490,9 @@ def measure_levels(r: StageResult, s: Strata, preset: dict, context: int, levels
|
||||
if stt:
|
||||
busy = stt.finish()
|
||||
row["stt"].update({k: busy.get(k) for k in ("status", "batch10_s", "single_s", "errors",
|
||||
"seconds")})
|
||||
for err in stt_gaps(busy):
|
||||
"seconds", "busy_share", "final_wait_s")
|
||||
if k in busy})
|
||||
for err in stt_gaps(busy, lv.get("stt_pace")):
|
||||
r.error(f"level {lv['key']} stt", err)
|
||||
else:
|
||||
r.error(f"level {lv['key']}", "the placement did not start")
|
||||
@@ -476,6 +521,64 @@ def measure_levels(r: StageResult, s: Strata, preset: dict, context: int, levels
|
||||
f"{most['key']}: {most['vram_free_mib']} MiB")
|
||||
|
||||
|
||||
def stt_growth_mib(work: Path) -> int:
|
||||
"""What ten speakers at once add over the loaded STT model, from this PC's own stt stage."""
|
||||
try:
|
||||
return int(read_json(work / "results" / "stt.json")["run_int8_float16"]["batch_speakers"]["vram_added_mib"])
|
||||
except (OSError, ValueError, KeyError, TypeError):
|
||||
return STT_GROWTH_FALLBACK_MIB
|
||||
|
||||
|
||||
def staging_gib(lines: list[str], default: float = 1.10) -> float:
|
||||
"""The long-prompt staging the expert budget holds first (engine log: "(1.10 GiB staging)")."""
|
||||
for line in lines:
|
||||
m = re.search(r"([0-9.]+) GiB staging", line)
|
||||
if m:
|
||||
return float(m.group(1))
|
||||
return default
|
||||
|
||||
|
||||
def expert_budget_gib(total_mib: int, used_mib: int, probe_model_mib: int, staging: float, growth_mib: int,
|
||||
margin_mib: int = STT_ROOM_MARGIN_MIB) -> float:
|
||||
"""--expert-vram (GiB, rounded down to 0.1) that leaves the STT's growth and the margin free.
|
||||
The budget first holds the staging, which the start without experts already has."""
|
||||
spare = total_mib - used_mib - probe_model_mib - growth_mib - margin_mib
|
||||
if spare <= 0:
|
||||
return 0.0
|
||||
return math.floor((staging * 1024 + spare) / 1024 * 10) / 10
|
||||
|
||||
|
||||
def stt_room(r: StageResult, s: Strata, preset: dict, row: dict, lv: dict, base: str) -> dict:
|
||||
"""Start the placement once without experts (the STT model already loaded), see what the LLM
|
||||
holds, stop it, and work out the expert budget that leaves the STT's ten-speaker growth free."""
|
||||
room: dict = {"stt_growth_mib": stt_growth_mib(s.log.parent.parent), "margin_mib": STT_ROOM_MARGIN_MIB,
|
||||
"expert_vram_gib": 0.0}
|
||||
say(" STT 자리 계산: 전문가 층 없이 한 번 켜서 LLM 기본 VRAM을 잽니다")
|
||||
before = gpu_memory_mib()
|
||||
launch = s.launch(preset, row["context"], parallel=lv["parallel"], vision=lv["vision"],
|
||||
extra_sets=["expert-vram=0"] + [x for x in lv["sets"] if not x.startswith("expert-vram=")],
|
||||
timeout=3600)
|
||||
room["probe_ready"] = bool(launch.get("ready"))
|
||||
if launch.get("ready"):
|
||||
try:
|
||||
ChatClient(base).chat([{"role": "user", "content": "안녕"}], max_tokens=8)
|
||||
except Exception as e:
|
||||
r.error(f"level {lv['key']} stt room", e)
|
||||
after = gpu_memory_mib()
|
||||
lines = s.engine_lines()
|
||||
if before and after:
|
||||
room.update(probe_model_mib=after[0] - before[0], used_before_mib=before[0], total_mib=before[1],
|
||||
staging_gib=staging_gib(lines))
|
||||
room["expert_vram_gib"] = expert_budget_gib(before[1], before[0], room["probe_model_mib"],
|
||||
room["staging_gib"], room["stt_growth_mib"])
|
||||
if room["expert_vram_gib"] == 0:
|
||||
r.note(f"{lv['key']}: no room for experts next to the STT; started with --expert-vram 0")
|
||||
else:
|
||||
r.error(f"level {lv['key']} stt room", "the start without experts did not come up; --expert-vram 0")
|
||||
s.stop()
|
||||
return room
|
||||
|
||||
|
||||
def ask_for_a_game() -> bool:
|
||||
"""The live stage needs someone to start a game first. False when nobody is there or they say no."""
|
||||
if not interactive():
|
||||
@@ -540,7 +643,8 @@ def measure_strata(r: StageResult, work: Path, which: str, context: int, only: s
|
||||
r["skipped"] = "no game started"
|
||||
return
|
||||
r.note("measured while the user's game was running")
|
||||
measure_levels(r, s, preset, context, LEVEL_SETS[only], stt_python=stt_python, choose=only != "live")
|
||||
measure_levels(r, s, preset, context, LEVEL_SETS[only], stt_python=stt_python,
|
||||
choose=only in ("levels", "together"))
|
||||
if only == "live":
|
||||
say("\a 게임 중 측정이 끝났습니다. 게임을 꺼도 됩니다.")
|
||||
try:
|
||||
@@ -618,7 +722,7 @@ def main() -> int:
|
||||
context = int(opts.get("context", "32768"))
|
||||
only = opts.get("only", "")
|
||||
name = {"tools": f"tools_{which}", "levels": "gamemode", "together": "gamemode_stt",
|
||||
"live": "gamemode_live"}.get(only, f"llm_{which}")
|
||||
"live": "gamemode_live", "tune": "gamemode_tune"}.get(only, f"llm_{which}")
|
||||
r = StageResult(name, out)
|
||||
try:
|
||||
if opts.get("url"):
|
||||
|
||||
@@ -7,6 +7,9 @@ With --opt mode=load it is instead the STT load the gamemode stages put next to
|
||||
the LLM: load the model on device=cuda|cpu, write {"status": "loaded"} to --out,
|
||||
wait for the file go=PATH, then transcribe ten speakers at once (and one final
|
||||
clip) on a loop until the file stop=PATH appears, and write the timings.
|
||||
partial_every=SECONDS|none and final_every=SECONDS pace that loop the way the app
|
||||
would while the LLM answers: a ten-speaker partial pass every N seconds (none: no
|
||||
partials) and a final clip every M seconds in between.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -77,27 +80,73 @@ def batch_transcribe(model, audios: list, beam: int = 1) -> tuple[list[str], flo
|
||||
return texts, time.perf_counter() - t0
|
||||
|
||||
|
||||
def load_loop(batch, single, go: Path, stop: Path, wait_s: float = 3600, max_s: float = 600) -> dict:
|
||||
"""Wait for ``go``, then run ``batch()`` (ten speakers) and ``single()`` (one final clip)
|
||||
in turn until ``stop`` appears. An error (out of memory) ends the loop and is kept."""
|
||||
def load_loop(batch, single, go: Path, stop: Path, pace: dict | None = None, wait_s: float = 3600,
|
||||
max_s: float = 600) -> dict:
|
||||
"""Wait for ``go``, then run ``batch()`` (a ten-speaker partial pass) and ``single()`` (one final
|
||||
clip) until ``stop`` appears. Without ``pace`` they run back to back (every speaker talking
|
||||
without a break). With ``pace`` a partial pass starts every ``partial_every`` s (None: none) and a
|
||||
final every ``final_every`` s. A final that is due goes first; one that falls due during a
|
||||
partial pass waits for it (final_wait_s).
|
||||
An error (out of memory) ends the loop and is kept."""
|
||||
t0 = time.monotonic()
|
||||
while not go.exists():
|
||||
if stop.exists() or time.monotonic() - t0 > wait_s:
|
||||
return {"status": "not_started", "batch10_s": stats([]), "single_s": stats([]), "errors": []}
|
||||
return {"status": "not_started", "batch10_s": stats([]), "single_s": stats([]), "errors": [],
|
||||
"pace": pace}
|
||||
time.sleep(0.2)
|
||||
b, one, errors = [], [], []
|
||||
t1 = time.monotonic()
|
||||
while not stop.exists() and time.monotonic() - t1 < max_s:
|
||||
# perf_counter: time.monotonic ticks every ~16 ms on Windows, too coarse for a 0.1 s clip.
|
||||
b, one, waits, errors = [], [], [], []
|
||||
t1 = time.perf_counter()
|
||||
every_p = (pace or {}).get("partial_every")
|
||||
every_f = (pace or {}).get("final_every")
|
||||
next_p, next_f = t1, t1
|
||||
|
||||
def timed(fn, into):
|
||||
s0 = time.perf_counter()
|
||||
fn()
|
||||
into.append(time.perf_counter() - s0)
|
||||
|
||||
while not stop.exists() and time.perf_counter() - t1 < max_s:
|
||||
try:
|
||||
for fn, into in ((batch, b), (single, one)):
|
||||
s0 = time.perf_counter()
|
||||
fn()
|
||||
into.append(time.perf_counter() - s0)
|
||||
if pace is None:
|
||||
timed(batch, b)
|
||||
timed(single, one)
|
||||
continue
|
||||
now = time.perf_counter()
|
||||
# A final goes first: the answer waits for it, not for the subtitles.
|
||||
if every_f is not None and now >= next_f:
|
||||
waits.append(now - next_f)
|
||||
next_f = max(next_f + every_f, now)
|
||||
timed(single, one)
|
||||
continue
|
||||
if every_p is not None and now >= next_p:
|
||||
next_p = now + every_p
|
||||
timed(batch, b)
|
||||
continue
|
||||
dues = [x for x, on in ((next_p, every_p is not None), (next_f, every_f is not None)) if on]
|
||||
time.sleep(min(max(min(dues, default=now + 0.1) - now, 0.0), 0.1))
|
||||
except Exception as e:
|
||||
errors.append(f"{type(e).__name__}: {str(e)[:300]}")
|
||||
break
|
||||
return {"status": "done", "batch10_s": stats(b), "single_s": stats(one), "errors": errors,
|
||||
"seconds": round(time.monotonic() - t1, 1)}
|
||||
seconds = time.perf_counter() - t1
|
||||
res = {"status": "done", "batch10_s": stats(b), "single_s": stats(one), "errors": errors,
|
||||
"seconds": round(seconds, 1), "pace": pace,
|
||||
"busy_share": round(min((sum(b) + sum(one)) / seconds, 1.0), 3) if seconds > 0 else None}
|
||||
if pace is not None:
|
||||
res["final_wait_s"] = stats(waits)
|
||||
return res
|
||||
|
||||
|
||||
def parse_pace(opts: dict) -> dict | None:
|
||||
"""partial_every=SECONDS|none, final_every=SECONDS; neither: back to back (None)."""
|
||||
if "partial_every" not in opts and "final_every" not in opts:
|
||||
return None
|
||||
|
||||
def num(key):
|
||||
v = opts.get(key, "none")
|
||||
return None if v == "none" else float(v)
|
||||
|
||||
return {"partial_every": num("partial_every"), "final_every": num("final_every")}
|
||||
|
||||
|
||||
def load_worker(out: Path, work: Path, opts: dict) -> int:
|
||||
@@ -122,7 +171,7 @@ def load_worker(out: Path, work: Path, opts: dict) -> int:
|
||||
return 0
|
||||
write_json(out, dict(res, status="loaded"))
|
||||
res.update(load_loop(lambda: batch_transcribe(model, clips), lambda: transcribe(model, clips[0], 1),
|
||||
Path(opts["go"]), Path(opts["stop"])))
|
||||
Path(opts["go"]), Path(opts["stop"]), pace=parse_pace(opts)))
|
||||
write_json(out, res)
|
||||
return 0
|
||||
|
||||
|
||||
5
scripts/m0/run_m0_gamemode_tune.bat
Normal file
5
scripts/m0/run_m0_gamemode_tune.bat
Normal file
@@ -0,0 +1,5 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0run_m0.ps1" -Only gamemode_tune -Redo %*
|
||||
echo.
|
||||
pause
|
||||
@@ -493,7 +493,7 @@ def _run_levels_stage(tmp_path, monkeypatch, fake_server, gpu, ready=lambda sets
|
||||
monkeypatch.setattr(llm, "gpu_memory_mib", lambda: gpu(running["now"]))
|
||||
monkeypatch.setattr(common, "gpu_memory_mib", lambda: gpu(running["now"]))
|
||||
out = tmp_path / "results" / "gamemode.json"
|
||||
out.parent.mkdir(parents=True)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
opts = ["--opt", "model=fast", "--opt", f"only={only}"]
|
||||
for o in extra_opts:
|
||||
opts += ["--opt", o]
|
||||
@@ -504,13 +504,14 @@ def _run_levels_stage(tmp_path, monkeypatch, fake_server, gpu, ready=lambda sets
|
||||
|
||||
def _model_mib(spec):
|
||||
"""Roughly what the 2026-11-10 game PC showed: 4.4 GiB of weights that never leave the card, the
|
||||
expert budget, the conversation memory (int8, 1.43 GiB per 32K; q4_0 half), about 1 GB for the
|
||||
MTP drafter and 1.5 GB of buffers. With no expert budget Strata fills the card."""
|
||||
experts = next((int(s.split("=")[1]) for s in spec["sets"] if s.startswith("expert-vram=")), None)
|
||||
expert budget (which first holds the 1.1 GiB long-prompt staging, so 0 and 1 cost the same), the
|
||||
conversation memory (int8, 1.43 GiB per 32K; q4_0 half), about 1 GB for the MTP drafter and
|
||||
1.5 GB of buffers. With no expert budget Strata fills the card."""
|
||||
experts = next((float(s.split("=")[1]) for s in spec["sets"] if s.startswith("expert-vram=")), None)
|
||||
if experts is None:
|
||||
return 13400
|
||||
kv = 1464 * spec["context"] / 32768 * (0.5 if spec["kv"] == "q4_0" else 1)
|
||||
return int(4506 + experts * 1024 + kv + (0 if spec["mtp"] == "off" else 1000) + 1500)
|
||||
return int(4506 + max(experts, 1.1) * 1024 + kv + (0 if spec["mtp"] == "off" else 1000) + 1500)
|
||||
|
||||
|
||||
def _vram(running):
|
||||
@@ -655,27 +656,35 @@ def test_normal_placement_answers_one_request_at_a_time(tmp_path, monkeypatch, f
|
||||
# --- gamemode_stt / gamemode_live: the LLM next to the STT that runs with it --
|
||||
|
||||
|
||||
STT_GPU_MIB = 2740 # M0 stt: the model 1300 MiB and 1440 MiB more for ten speakers at once
|
||||
STT_MODEL_MIB = 1300 # M0 stt: the loaded model
|
||||
STT_GROWTH_MIB = 1440 # M0 stt: what ten speakers at once add over the loaded model
|
||||
STT_GPU_MIB = STT_MODEL_MIB + STT_GROWTH_MIB
|
||||
|
||||
|
||||
class _FakeStt:
|
||||
"""Stands in for the STT worker process: on cuda its model holds VRAM while it is loaded."""
|
||||
"""Stands in for the STT worker process: on cuda its model holds VRAM while it is loaded, and
|
||||
ten speakers at once hold more once it is transcribing (with partials on)."""
|
||||
|
||||
def __init__(self, fail_device=None, broken_at_load=False, finish_result=None):
|
||||
self.finish_result = finish_result
|
||||
self.events: list[str] = []
|
||||
self.devices: list[str] = []
|
||||
self.paces: list = []
|
||||
self.loaded: str | None = None
|
||||
self.busy = False
|
||||
self.running: dict = {}
|
||||
self.fail_device = fail_device
|
||||
self.broken_at_load = broken_at_load
|
||||
|
||||
def used_mib(self) -> int:
|
||||
return STT_GPU_MIB if self.loaded == "cuda" else 0
|
||||
if self.loaded != "cuda":
|
||||
return 0
|
||||
return STT_GPU_MIB if self.busy else STT_MODEL_MIB
|
||||
|
||||
def factory(self, python, work, device, log):
|
||||
def factory(self, python, work, device, log, pace=None):
|
||||
fake = self
|
||||
fake.devices.append(device)
|
||||
fake.paces.append(pace)
|
||||
|
||||
class _One:
|
||||
def start(self, timeout=900):
|
||||
@@ -688,17 +697,20 @@ class _FakeStt:
|
||||
def go(self):
|
||||
assert fake.running.get("now") is not None, "STT load must overlap a running LLM"
|
||||
fake.events.append("stt_go")
|
||||
fake.busy = pace is None or pace.get("partial_every") is not None
|
||||
|
||||
def finish(self, timeout=120):
|
||||
fake.events.append("stt_finish")
|
||||
if fake.finish_result is not None:
|
||||
return fake.finish_result
|
||||
errs = ["RuntimeError: CUDA out of memory"] if device == fake.fail_device else []
|
||||
return {"status": "done", "batch10_s": {"n": 4, "median": 0.9}, "single_s": {"n": 4, "median": 0.2},
|
||||
"errors": errs}
|
||||
partials = pace is None or pace.get("partial_every") is not None
|
||||
return {"status": "done", "batch10_s": {"n": 4 if partials else 0, "median": 0.9},
|
||||
"single_s": {"n": 4, "median": 0.2}, "errors": errs, "pace": pace}
|
||||
|
||||
def close(self):
|
||||
fake.loaded = None
|
||||
fake.busy = False
|
||||
|
||||
return _One()
|
||||
|
||||
@@ -883,3 +895,129 @@ def test_stt_worker_that_ignores_stop_is_reported_as_timed_out_and_killed(tmp_pa
|
||||
assert res["status"] == "timeout" and res["errors"]
|
||||
s.close()
|
||||
assert s.proc.poll() is not None
|
||||
|
||||
|
||||
# --- gamemode_tune: room for the STT in the normal placement, fewer partials while the LLM answers --
|
||||
|
||||
|
||||
class _FakeClock:
|
||||
"""time.monotonic/perf_counter/sleep for the STT loop: passes advance it, nothing waits."""
|
||||
|
||||
def __init__(self):
|
||||
self.t = 0.0
|
||||
|
||||
def monotonic(self):
|
||||
return self.t
|
||||
|
||||
perf_counter = monotonic
|
||||
|
||||
def sleep(self, s):
|
||||
self.t += s
|
||||
|
||||
|
||||
def _paced_run(tmp_path, monkeypatch, pace, run_s=10.0):
|
||||
from m0lib import stt
|
||||
clock = _FakeClock()
|
||||
monkeypatch.setattr(stt, "time", clock)
|
||||
go, stop = tmp_path / "go", tmp_path / "stop"
|
||||
go.write_text("")
|
||||
|
||||
def work(seconds):
|
||||
def fn():
|
||||
clock.t += seconds
|
||||
if clock.t >= run_s:
|
||||
stop.write_text("")
|
||||
return fn
|
||||
|
||||
return stt.load_loop(work(0.9), work(0.2), go, stop, pace=pace)
|
||||
|
||||
|
||||
def test_paced_stt_runs_partials_at_their_interval_and_finals_in_between(tmp_path, monkeypatch):
|
||||
res = _paced_run(tmp_path, monkeypatch, {"partial_every": 2.0, "final_every": 0.5})
|
||||
assert res["status"] == "done" and res["pace"] == {"partial_every": 2.0, "final_every": 0.5}
|
||||
assert 5 <= res["batch10_s"]["n"] <= 6 # one ten-speaker pass every 2 s over 10 s
|
||||
assert res["single_s"]["n"] >= 12 # finals keep coming in between
|
||||
# A final waits at most for the partial pass that is running when it is due.
|
||||
assert res["final_wait_s"]["max"] <= 0.9 + 1e-6
|
||||
assert res["busy_share"] < 1.0 # the GPU gets time back for the LLM
|
||||
|
||||
|
||||
def test_stt_without_a_pace_transcribes_back_to_back(tmp_path, monkeypatch):
|
||||
res = _paced_run(tmp_path, monkeypatch, None)
|
||||
assert res["busy_share"] == 1.0 and res["batch10_s"]["n"] == res["single_s"]["n"]
|
||||
|
||||
|
||||
def test_finals_only_stt_runs_no_ten_speaker_pass(tmp_path, monkeypatch):
|
||||
res = _paced_run(tmp_path, monkeypatch, {"partial_every": None, "final_every": 0.5})
|
||||
assert res["batch10_s"]["n"] == 0 and res["single_s"]["n"] >= 15
|
||||
assert res["busy_share"] < 0.5
|
||||
|
||||
|
||||
def test_stt_worker_command_carries_the_pace(tmp_path):
|
||||
s = llm.SttLoad(sys.executable, tmp_path, "cuda", tmp_path / "stt.log",
|
||||
pace={"partial_every": None, "final_every": 0.5})
|
||||
assert "partial_every=none" in s.cmd and "final_every=0.5" in s.cmd
|
||||
plain = llm.SttLoad(sys.executable, tmp_path, "cuda", tmp_path / "stt2.log")
|
||||
assert not any(str(x).startswith(("partial_every=", "final_every=")) for x in plain.cmd)
|
||||
|
||||
|
||||
def _expert_vram(spec):
|
||||
return next((float(s.split("=")[1]) for s in spec["sets"] if s.startswith("expert-vram=")), None)
|
||||
|
||||
|
||||
def test_tune_stage_leaves_the_stt_room_in_the_normal_placement(tmp_path, monkeypatch, fake_server):
|
||||
stt = _FakeStt()
|
||||
r, started, running = _run_levels_stage(tmp_path, monkeypatch, fake_server, _vram, only="tune",
|
||||
extra_opts=["stt_python=py"], stt=stt)
|
||||
assert r["stage"] == "gamemode_tune" and r["status"] == "ok"
|
||||
normal = [lv for lv in llm.TUNED if lv.get("stt_room")]
|
||||
# One start without experts measures what the LLM needs before any expert, with the STT loaded.
|
||||
probes = [x for x in started if _expert_vram(x) == 0 and x["vision"] == normal[0]["vision"]]
|
||||
assert len(probes) == 1 and started[0] is probes[0]
|
||||
rows = {x["key"]: x for x in r["levels"]}
|
||||
budgets = {_expert_vram(x) for x in started[1:] if x["vision"] == normal[0]["vision"]}
|
||||
assert len(budgets) == 1 and budgets.pop() > 0
|
||||
# With the STT at its ten-speaker peak the card keeps the margin free, and not much more.
|
||||
continuous = [rows[lv["key"]] for lv in normal if lv.get("stt_pace") is None]
|
||||
assert continuous
|
||||
for row in continuous:
|
||||
assert llm.STT_ROOM_MARGIN_MIB <= row["vram_free_mib"] < llm.STT_ROOM_MARGIN_MIB + 2 * 103
|
||||
assert row["stt_room"]["stt_growth_mib"] == llm.STT_GROWTH_FALLBACK_MIB
|
||||
# The paces: back to back, fewer partials, finals only; in the normal and the 2 GB game placement.
|
||||
paces = {(lv.get("stt_pace") or {}).get("partial_every", "continuous") for lv in normal}
|
||||
assert paces == {"continuous", 2.0, None}
|
||||
game = [lv for lv in llm.TUNED if lv["key"].startswith("game_")]
|
||||
assert game and all("expert-vram=2" in lv["sets"] for lv in game)
|
||||
assert all(lv.get("stt_pace") for lv in game)
|
||||
assert stt.paces == [lv.get("stt_pace") for lv in llm.TUNED]
|
||||
assert stt.loaded is None and running["now"] is None
|
||||
|
||||
|
||||
def test_stt_room_comes_from_this_pcs_own_stt_measurement(tmp_path, monkeypatch, fake_server):
|
||||
(tmp_path / "results").mkdir(parents=True)
|
||||
(tmp_path / "results" / "stt.json").write_text(json.dumps(
|
||||
{"run_int8_float16": {"batch_speakers": {"vram_added_mib": STT_GROWTH_MIB + 512}}}), encoding="utf-8")
|
||||
r, started, _ = _run_levels_stage(tmp_path, monkeypatch, fake_server, _vram, only="tune",
|
||||
extra_opts=["stt_python=py"], stt=_FakeStt())
|
||||
row = next(x for x in r["levels"] if x.get("stt_room"))
|
||||
assert row["stt_room"]["stt_growth_mib"] == STT_GROWTH_MIB + 512
|
||||
assert row["vram_free_mib"] >= llm.STT_ROOM_MARGIN_MIB + 512
|
||||
|
||||
|
||||
def test_card_without_room_for_any_expert_starts_the_normal_placement_with_none(tmp_path, monkeypatch,
|
||||
fake_server):
|
||||
def small(running):
|
||||
return _vram(running)[0], 12000
|
||||
|
||||
r, started, _ = _run_levels_stage(tmp_path, monkeypatch, fake_server, small, only="tune",
|
||||
extra_opts=["stt_python=py"], stt=_FakeStt())
|
||||
row = next(x for x in r["levels"] if x.get("stt_room"))
|
||||
assert row["stt_room"]["expert_vram_gib"] == 0
|
||||
assert any("no room" in n for n in r["notes"])
|
||||
|
||||
|
||||
def test_tune_stage_runs_with_the_stt_python_and_has_a_one_click_runner(tmp_path, monkeypatch):
|
||||
import m0
|
||||
assert "gamemode_tune" in m0.NEEDS_STT and "gamemode_tune" not in m0.ON_DEMAND
|
||||
bat = (Path(m0.KIT) / "run_m0_gamemode_tune.bat").read_text(encoding="utf-8")
|
||||
assert "-Only gamemode_tune" in bat and "-Redo" in bat
|
||||
|
||||
Reference in New Issue
Block a user