diff --git a/docs/v2/roadmap.md b/docs/v2/roadmap.md index 7902618..9a4b79b 100644 --- a/docs/v2/roadmap.md +++ b/docs/v2/roadmap.md @@ -4,7 +4,7 @@ | 단계 | 내용 | 완료 기준 | |---|---|---| -| M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. | +| M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. 측정 키트: [scripts/m0/](../../scripts/m0/README.md) | | M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과 | | M2 | 디스코드 음성 코어 이전 | v1 봇 코드를 Electron 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기) | | M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과 | diff --git a/scripts/m0/README.md b/scripts/m0/README.md new file mode 100644 index 0000000..ebc74ec --- /dev/null +++ b/scripts/m0/README.md @@ -0,0 +1,54 @@ +# M0 측정 키트 (게임PC 사전 검증) + +v2 기본값을 정하려고 게임PC에서 한 번 실행하는 측정 스크립트입니다. 결과는 zip 파일 하나로 나오고, 그 파일을 디스코드로 보내 주시면 됩니다. + +## 실행 방법 + +1. 이 폴더(`jarvis_m0`)를 **남은 공간이 가장 많은 드라이브**에 풀어 둡니다. 예: `D:\jarvis_m0` + - 작업 폴더는 같은 드라이브의 `\JarvisM0`(예: `D:\JarvisM0`)에 만들어집니다. + - LLM 모델 두 개를 받으므로 **약 150GB**가 필요합니다. 빠름 모델만 재면 약 50GB입니다. +2. `run_m0.bat`을 더블클릭합니다. + - Windows가 "PC 보호" 경고를 띄우면 "추가 정보 → 실행"을 누릅니다. +3. 처음 몇 분은 화면 안내에 따라 직접 답해 주셔야 합니다(y/n). 디스코드 앱을 켜 두세요. +4. "이제 자리를 비우셔도 됩니다"가 나오면 그 뒤는 자동입니다. 모델 다운로드 때문에 몇 시간 걸릴 수 있습니다. **측정이 끝날 때까지 게임과 무거운 프로그램은 꺼 두세요.** +5. 끝나면 `jarvis_m0` 폴더에 `M0_results_날짜_시각.zip`이 생기고 탐색기에서 그 파일이 선택됩니다. 이 파일을 보내 주세요. + +중간에 꺼져도 `run_m0.bat`을 다시 실행하면 끝난 단계는 건너뛰고 이어서 합니다. + +## 직접 답하는 부분 + +| 확인 | 하실 일 | +|---|---| +| 봇용 창 숨김 | 시계가 움직이는 테스트 창이 뜹니다. Alt+Tab, 작업 표시줄, 디스코드 "화면 공유 → 창" 목록에 보이는지 답합니다. | +| 디스코드 창 분리 | 친구(또는 다른 기기의 내 계정)의 화면공유를 시청하고 "창 분리(팝아웃)"를 누릅니다. 지금 할 수 없으면 `s`로 건너뜁니다. | +| 게임 감지 | 게임을 하나 켜고(테두리 없는 창 모드면 가장 좋음), 안티치트 게임이 있으면 그것도 켭니다. 마지막으로 브라우저 영상을 전체화면으로 틉니다. 각각 Enter 후 15초 안에 그 화면으로 바꾸면 됩니다. | + +## 자동으로 재는 것 + +| 단계 | 내용 | +|---|---| +| PC 정보 | GPU, 드라이버, CPU, RAM, 디스크, 모니터 | +| MeloTTS | 한국어 문장 합성 시간, VRAM | +| CosyVoice2 / CosyVoice3 | 한국어 첫 소리까지 시간, 합성 속도, VRAM | +| STT | faster-whisper large-v3-turbo의 전사 시간, 정확도, 말하는 중 부분 전사, 10명 동시 처리 | +| LLM 빠름 / 권장 | Strata로 Qwen3.6-35B-A3B(약 23GB)와 Qwen3.8-Flash-Next Q2_0(약 66GB)을 받아서 첫 토큰 시간(맥락 길이별), 생성 속도, 도구 호출, 이미지 읽기(화질 단계별), 한국어 답변, 동시 요청, 게임 모드용 VRAM 비우기와 모델 다시 시작·전환 시간 | + +## 개인정보 + +- 모든 측정은 PC 안에서 합니다. 인터넷은 프로그램과 모델을 받을 때만 씁니다. +- 결과 파일에서 사용자 이름, 컴퓨터 이름, 사용자 폴더 경로는 가립니다. +- 다른 프로그램의 창 제목은 저장하지 않습니다. 디스코드 분리 창 제목은 저장해도 되는지 물어본 뒤에만 넣습니다. +- 게임 감지는 실행 중인 프로그램 이름과 창 모양만 봅니다. 게임 화면은 저장하지 않습니다. 설치된 게임은 이름과 안티치트 유무만 기록합니다. +- 음성 테스트는 키트에 들어 있는 합성 음성만 씁니다. 마이크는 쓰지 않습니다. + +## 옵션 (PowerShell에서 직접 실행할 때) + +```powershell +.\run_m0.ps1 -WorkDir E:\JarvisM0 # 작업 폴더 위치 지정 +.\run_m0.ps1 -Skip llm_recommended # 권장 모델(66GB) 빼고 실행 +.\run_m0.ps1 -Only stt,tts_melo -Redo # 특정 단계만 다시 +``` + +단계 이름: `sysinfo`, `winchecks`, `tts_melo`, `tts_cosy2`, `tts_cosy3`, `stt`, `llm_fast`, `llm_recommended` + +다 끝나고 필요 없으면 작업 폴더(`\JarvisM0`)를 통째로 지우면 됩니다. Python, 모델, Strata(엔진과 설정 포함)는 모두 그 안에 설치하도록 지정해 두었습니다. diff --git a/scripts/m0/assets/ko_samples/ko_01.wav b/scripts/m0/assets/ko_samples/ko_01.wav new file mode 100644 index 0000000..02d02f4 Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_01.wav differ diff --git a/scripts/m0/assets/ko_samples/ko_02.wav b/scripts/m0/assets/ko_samples/ko_02.wav new file mode 100644 index 0000000..b829581 Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_02.wav differ diff --git a/scripts/m0/assets/ko_samples/ko_03.wav b/scripts/m0/assets/ko_samples/ko_03.wav new file mode 100644 index 0000000..8640d23 Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_03.wav differ diff --git a/scripts/m0/assets/ko_samples/ko_04.wav b/scripts/m0/assets/ko_samples/ko_04.wav new file mode 100644 index 0000000..ae6e92b Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_04.wav differ diff --git a/scripts/m0/assets/ko_samples/ko_05.wav b/scripts/m0/assets/ko_samples/ko_05.wav new file mode 100644 index 0000000..736a3ff Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_05.wav differ diff --git a/scripts/m0/assets/ko_samples/ko_06.wav b/scripts/m0/assets/ko_samples/ko_06.wav new file mode 100644 index 0000000..68aa3a6 Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_06.wav differ diff --git a/scripts/m0/assets/ko_samples/ko_07.wav b/scripts/m0/assets/ko_samples/ko_07.wav new file mode 100644 index 0000000..cb53898 Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_07.wav differ diff --git a/scripts/m0/assets/ko_samples/ko_08.wav b/scripts/m0/assets/ko_samples/ko_08.wav new file mode 100644 index 0000000..18857c0 Binary files /dev/null and b/scripts/m0/assets/ko_samples/ko_08.wav differ diff --git a/scripts/m0/assets/ko_samples/refs.json b/scripts/m0/assets/ko_samples/refs.json new file mode 100644 index 0000000..598d19a --- /dev/null +++ b/scripts/m0/assets/ko_samples/refs.json @@ -0,0 +1,45 @@ +{ + "source": "MeloTTS KR speed 1.0, 16 kHz mono (synthetic)", + "samples": [ + { + "file": "ko_01.wav", + "text": "자비스, 오늘 서울 날씨 어때?", + "seconds": 4.02 + }, + { + "file": "ko_02.wav", + "text": "자비스, 내일 아침 일곱 시에 알람 맞춰 줘.", + "seconds": 5.0 + }, + { + "file": "ko_03.wav", + "text": "지금 두 번째 모니터에 뭐 떠 있는지 알려 줄래?", + "seconds": 4.83 + }, + { + "file": "ko_04.wav", + "text": "어제 저녁에 먹은 김치찌개랑 공기밥 기록해 줘.", + "seconds": 5.09 + }, + { + "file": "ko_05.wav", + "text": "자비스, 이번 주말에 볼 만한 영화 추천해 줘.", + "seconds": 4.85 + }, + { + "file": "ko_06.wav", + "text": "철수 화면공유에서 지금 무슨 게임 하고 있어?", + "seconds": 4.74 + }, + { + "file": "ko_07.wav", + "text": "잠깐만, 그건 됐고 음악 좀 틀어 줘.", + "seconds": 4.71 + }, + { + "file": "ko_08.wav", + "text": "오늘 회의는 오후 세 시에 시작해서 다섯 시쯤 끝날 것 같아요. 끝나고 나서 저녁 메뉴를 같이 정해 봐요.", + "seconds": 9.83 + } + ] +} \ No newline at end of file diff --git a/scripts/m0/m0.py b/scripts/m0/m0.py new file mode 100644 index 0000000..746a27d --- /dev/null +++ b/scripts/m0/m0.py @@ -0,0 +1,158 @@ +"""Jarvis v2 M0: measure the gaming PC once and pack the results into one zip. + +Started by run_m0.bat -> run_m0.ps1 (which installs uv and Python into the +work folder). Each stage writes results/.json; a stage that already has +a result is skipped on the next run, so the script can be started again after +an interruption. +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import os +import shutil +import subprocess +import sys +import time +import zipfile +from pathlib import Path + +KIT = Path(__file__).resolve().parent +sys.path.insert(0, str(KIT)) +from m0lib import envs # noqa: E402 +from m0lib.common import read_json, say, step, write_json # noqa: E402 + +# key, title, venv ("" = this Python), script, options, free disk needed (GB) +STAGES = [ + ("sysinfo", "PC 정보", "", "sysinfo.py", [], 0), + ("winchecks", "창·디스코드·게임 확인 (직접 답해 주셔야 합니다)", "", "winchecks.py", [], 0), + ("tts_melo", "MeloTTS 속도", "melo", "tts_melo.py", [], 10), + ("tts_cosy2", "CosyVoice2 속도", "cosy", "tts_cosy.py", ["models=cosyvoice2"], 15), + ("tts_cosy3", "Fun-CosyVoice3 속도", "cosy", "tts_cosy.py", ["models=cosyvoice3"], 8), + ("stt", "STT(faster-whisper turbo) 속도", "stt", "stt.py", [], 5), + ("llm_fast", "LLM 빠름 모델 (Qwen3.6-35B-A3B, 약 23GB 받기)", "base", "llm.py", ["model=fast"], 35), + ("llm_recommended", "LLM 권장 모델 (Qwen3.8-Flash-Next Q2_0, 약 66GB 받기)", "base", "llm.py", + ["model=recommended"], 80), +] + + +def free_gb(path: Path) -> float: + return shutil.disk_usage(path).free / 1e9 + + +def run_stage(key: str, venv: str, script: str, opts: list[str], work: Path, env: dict) -> None: + out = work / "results" / f"{key}.json" + log = work / "logs" / f"env_{venv or 'main'}.log" + if venv: + py = str(envs.ensure_env(venv, work, log)) + else: + py = sys.executable + if venv == "cosy": + src = envs.ensure_cosy_source(work, log) + opts = opts + [f"src={src}"] + cmd = [py, str(KIT / "m0lib" / script), "--out", str(out), "--work", str(work)] + for o in opts: + cmd += ["--opt", o] + # Inherit the console so interactive stages can ask questions; stage + # scripts keep their own heavy output in logs/. + proc = subprocess.run(cmd, env=env) + if not out.exists(): + write_json(out, {"stage": key, "status": "failed", "errors": [ + {"where": "process", "error": f"stage exited with {proc.returncode} without a result"}]}) + + +def summarise(work: Path) -> str: + lines = ["# Jarvis v2 M0 결과 요약", "", f"만든 시각: {dt.datetime.now():%Y-%m-%d %H:%M}", ""] + for key, title, *_ in STAGES: + p = work / "results" / f"{key}.json" + if not p.exists(): + lines.append(f"- {title}: 실행 안 함") + continue + d = read_json(p) + errs = len(d.get("errors") or []) + lines.append(f"- {title}: {d.get('status')} ({d.get('seconds', '?')}초, 오류 {errs}개)") + lines += ["", "자세한 수치는 같은 zip 안의 results/*.json에 있습니다. 이 zip 파일을 그대로 보내 주시면 됩니다."] + return "\n".join(lines) + "\n" + + +def pack(work: Path) -> Path: + stamp = dt.datetime.now().strftime("%Y%m%d_%H%M") + target = KIT / f"M0_results_{stamp}.zip" + summary = summarise(work) + (work / "results" / "summary.md").write_text(summary, encoding="utf-8") + with zipfile.ZipFile(target, "w", zipfile.ZIP_DEFLATED) as z: + for p in sorted((work / "results").rglob("*")): + if p.is_file() and p.suffix in (".json", ".wav", ".md"): + z.write(p, f"results/{p.relative_to(work / 'results').as_posix()}") + for p in sorted((work / "logs").glob("*.log")): + data = p.read_bytes() + # Keep the end of long logs (download progress) so the zip stays small. + z.writestr(f"logs/{p.name}", data[-1_000_000:]) + return target + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--work", required=True) + ap.add_argument("--only", default="", help="comma list of stage keys") + ap.add_argument("--skip", default="", help="comma list of stage keys") + ap.add_argument("--redo", action="store_true", help="measure again even if a result exists") + ap.add_argument("--llm-url", default="", help="measure this OpenAI-compatible server instead of Strata") + a = ap.parse_args() + + work = Path(a.work).resolve() + for d in ("results", "logs", "cache/tmp"): + (work / d).mkdir(parents=True, exist_ok=True) + env = dict(os.environ, PYTHONUTF8="1", PYTHONIOENCODING="utf-8", HF_HOME=str(work / "models" / "hf"), + TEMP=str(work / "cache" / "tmp"), TMP=str(work / "cache" / "tmp"), + HF_HUB_DISABLE_TELEMETRY="1", DO_NOT_TRACK="1") + only = {x for x in a.only.split(",") if x} + skip = {x for x in a.skip.split(",") if x} + + say("Jarvis v2 M0 측정을 시작합니다.") + say(f"작업 폴더: {work} (남은 공간 {free_gb(work):.0f} GB)") + say("모든 파일은 이 작업 폴더 안에만 받습니다. 끝나면 결과 zip 파일 위치를 알려 드립니다.") + say("처음 두 단계는 직접 답해 주셔야 하고, 그 뒤는 자리를 비우셔도 됩니다.") + say("측정이 정확하도록 그동안 게임이나 무거운 프로그램은 꺼 두세요.\n") + + t0 = time.time() + for key, title, venv, script, opts, need_gb in STAGES: + if (only and key not in only) or key in skip: + continue + out = work / "results" / f"{key}.json" + if out.exists() and not a.redo: + status = read_json(out).get("status") + if status in ("ok", "partial", "skipped"): + say(f"[건너뜀] {title}: 이미 측정했습니다 ({status}). 다시 재려면 --redo") + continue + step(title) + if need_gb and free_gb(work) < need_gb: + say(f" 디스크 공간이 부족합니다 (필요 약 {need_gb} GB, 남은 {free_gb(work):.0f} GB). 건너뜁니다.") + write_json(out, {"stage": key, "status": "failed", + "errors": [{"where": "disk", "error": f"free {free_gb(work):.0f} GB < {need_gb} GB"}]}) + continue + stage_opts = list(opts) + if key.startswith("llm_") and a.llm_url: + stage_opts.append(f"url={a.llm_url}") + try: + run_stage(key, venv, script, stage_opts, work, env) + except Exception as e: + say(f" [오류] {title}: {e}") + write_json(out, {"stage": key, "status": "failed", "errors": [{"where": "setup", "error": str(e)}]}) + if key == "winchecks": + say("\n직접 확인하는 단계는 끝났습니다. 이제 자리를 비우셔도 됩니다.") + + target = pack(work) + say("\n" + summarise(work)) + say(f"걸린 시간: {(time.time() - t0) / 60:.0f}분") + say(f"결과 파일: {target}") + say("이 zip 파일을 디스코드로 보내 주세요.") + if os.name == "nt": + subprocess.run(["explorer", "/select,", str(target)]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/m0/m0lib/common.py b/scripts/m0/m0lib/common.py new file mode 100644 index 0000000..4ec87e4 --- /dev/null +++ b/scripts/m0/m0lib/common.py @@ -0,0 +1,321 @@ +"""Shared helpers for the M0 measurement kit (standard library only). + +Every stage script imports this module, both from the orchestrator's Python +and from the per-stage virtual environments, so it must not import anything +outside the standard library. +""" + +from __future__ import annotations + +import datetime as _dt +import json +import os +import re +import shutil +import statistics +import subprocess +import sys +import threading +import time +from pathlib import Path +from typing import Any, Callable, Iterable + +IS_WINDOWS = os.name == "nt" + + +def now_iso() -> str: + return _dt.datetime.now().astimezone().isoformat(timespec="seconds") + + +# -------------------------------------------------------------------------- +# Privacy: results leave the PC, so personal paths and names are masked. + +def _personal_tokens() -> list[str]: + tokens = [] + for key in ("USERNAME", "USER", "COMPUTERNAME", "USERDOMAIN"): + v = os.environ.get(key, "").strip() + if len(v) >= 2 and v.lower() not in ("root", "admin", "user", "workgroup"): + tokens.append(v) + home = str(Path.home()) + if home: + tokens.append(home) + # Longest first so a home path is masked before the bare user name. + return sorted(set(tokens), key=len, reverse=True) + + +_TOKENS = _personal_tokens() + + +def scrub(text: str) -> str: + """Mask the home folder, user name and computer name in ``text``.""" + if not text: + return text + out = text + for t in _TOKENS: + if "\\" in t or "/" in t: + out = out.replace(t, "") + out = out.replace(t.replace("\\", "/"), "") + out = out.replace(t.replace("\\", "\\\\"), "") + else: + # Whole words only: a short account name such as "ci" must not + # turn "insufficient" into "insuffient". + out = re.sub(rf"(?", out, flags=re.IGNORECASE) + return out + + +def scrub_obj(obj: Any) -> Any: + if isinstance(obj, str): + return scrub(obj) + if isinstance(obj, list): + return [scrub_obj(x) for x in obj] + if isinstance(obj, tuple): + return [scrub_obj(x) for x in obj] + if isinstance(obj, dict): + return {scrub(str(k)): scrub_obj(v) for k, v in obj.items()} + return obj + + +# -------------------------------------------------------------------------- +# Console output. Messages for the user are Korean; keep them short. + +_print_lock = threading.Lock() + + +def say(msg: str) -> None: + with _print_lock: + print(msg, flush=True) + + +def step(msg: str) -> None: + say(f"\n=== {msg} ===") + + +def interactive() -> bool: + """False when stdin is not a console or M0_AUTO=1 (unattended test runs).""" + return bool(sys.stdin) and sys.stdin.isatty() and os.environ.get("M0_AUTO") != "1" + + +def ask(question: str, choices: str = "yn", default: str | None = None) -> str: + """Ask a one-letter question. Returns the chosen letter (lower case). + + When not interactive the default is returned (or the first choice). + """ + fallback = default or choices[0] + if not interactive(): + say(f"{question} -> (자동 응답) {fallback}") + return fallback + hint = "/".join(choices) + while True: + try: + ans = input(f"{question} [{hint}] ").strip().lower() + except EOFError: + return fallback + if not ans and default: + return default + if ans and ans[0] in choices: + return ans[0] + say(f" {hint} 중 하나를 입력해 주세요.") + + +def wait_enter(msg: str) -> str: + """Wait for Enter. Returns 's' when the user typed s (skip).""" + if not interactive(): + say(f"{msg} -> (자동 진행)") + return "" + try: + return input(f"{msg} ").strip().lower()[:1] + except EOFError: + return "" + + +# -------------------------------------------------------------------------- +# Results and logs + +def write_json(path: Path, data: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(scrub_obj(data), ensure_ascii=False, indent=2), encoding="utf-8") + os.replace(tmp, path) + + +def read_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +class StageResult: + """Collects one stage's measurements and always writes them, even on error.""" + + def __init__(self, stage: str, out_path: Path): + self.stage = stage + self.out_path = out_path + self.data: dict[str, Any] = {"stage": stage, "started": now_iso(), "status": "running", + "errors": [], "notes": []} + self._t0 = time.perf_counter() + + def __setitem__(self, key: str, value: Any) -> None: + self.data[key] = value + + def __getitem__(self, key: str) -> Any: + return self.data[key] + + def get(self, key: str, default: Any = None) -> Any: + return self.data.get(key, default) + + def error(self, where: str, exc: BaseException | str) -> None: + text = exc if isinstance(exc, str) else f"{type(exc).__name__}: {exc}" + self.data["errors"].append({"where": where, "error": text[:2000]}) + say(f" [오류] {where}: {text[:300]}") + + def note(self, text: str) -> None: + self.data["notes"].append(text) + + def save(self, status: str | None = None) -> None: + if status: + self.data["status"] = status + self.data["seconds"] = round(time.perf_counter() - self._t0, 1) + self.data["finished"] = now_iso() + write_json(self.out_path, self.data) + + +def run_logged(cmd: list[str], log_path: Path, *, timeout: float | None = None, + env: dict[str, str] | None = None, cwd: str | None = None, + echo: bool = False, echo_filter: Callable[[str], bool] | None = None) -> int: + """Run ``cmd`` and append its combined output to ``log_path``. + + Returns the exit code, or -9 on timeout. With ``echo`` the lines are also + printed (optionally filtered) so long downloads show progress. + """ + log_path.parent.mkdir(parents=True, exist_ok=True) + with open(log_path, "a", encoding="utf-8", errors="replace") as log: + log.write(f"\n$ {scrub(' '.join(cmd))}\n") + log.flush() + proc = subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env, cwd=cwd, + text=True, encoding="utf-8", errors="replace", bufsize=1) + deadline = time.monotonic() + timeout if timeout else None + timed_out = threading.Event() + + def _watch() -> None: + while proc.poll() is None: + if deadline and time.monotonic() > deadline: + timed_out.set() + proc.kill() + return + time.sleep(1) + + threading.Thread(target=_watch, daemon=True).start() + assert proc.stdout is not None + for line in proc.stdout: + log.write(scrub(line)) + if echo and (echo_filter is None or echo_filter(line)): + say(" " + line.rstrip()[:200]) + proc.wait() + if timed_out.is_set(): + log.write(f"[timeout after {timeout}s]\n") + return -9 + log.write(f"[exit {proc.returncode}]\n") + return proc.returncode + + +# -------------------------------------------------------------------------- +# GPU sampling via nvidia-smi (present with every NVIDIA driver) + +def nvidia_smi() -> str | None: + exe = shutil.which("nvidia-smi") + if exe: + return exe + if IS_WINDOWS: + for p in (r"C:\Windows\System32\nvidia-smi.exe", + r"C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.exe"): + if os.path.exists(p): + return p + return None + + +def gpu_memory_mib() -> tuple[int, int] | None: + """(used, total) MiB of GPU 0, or None.""" + exe = nvidia_smi() + if not exe: + return None + try: + out = subprocess.run([exe, "--query-gpu=memory.used,memory.total", "--format=csv,noheader,nounits"], + capture_output=True, text=True, timeout=10).stdout.strip().splitlines()[0] + used, total = (int(x.strip()) for x in out.split(",")) + return used, total + except Exception: + return None + + +class GpuSampler: + """Samples GPU memory in the background; ``peak`` is the max used MiB.""" + + def __init__(self, interval: float = 0.5): + self.interval = interval + self.peak: int | None = None + self.start_used: int | None = None + self._stop = threading.Event() + self._t: threading.Thread | None = None + + def __enter__(self) -> "GpuSampler": + m = gpu_memory_mib() + self.start_used = m[0] if m else None + self.peak = self.start_used + self._t = threading.Thread(target=self._loop, daemon=True) + self._t.start() + return self + + def _loop(self) -> None: + while not self._stop.is_set(): + m = gpu_memory_mib() + if m and (self.peak is None or m[0] > self.peak): + self.peak = m[0] + self._stop.wait(self.interval) + + def __exit__(self, *exc: Any) -> None: + self._stop.set() + if self._t: + self._t.join(timeout=5) + + def summary(self) -> dict[str, Any]: + out = {"vram_used_before_mib": self.start_used, "vram_peak_mib": self.peak} + if self.start_used is not None and self.peak is not None: + out["vram_added_mib"] = self.peak - self.start_used + return out + + +# -------------------------------------------------------------------------- +# Small numeric helpers + +def stats(values: Iterable[float], digits: int = 3) -> dict[str, Any]: + v = [float(x) for x in values if x is not None] + if not v: + return {"n": 0} + return {"n": len(v), "median": round(statistics.median(v), digits), "min": round(min(v), digits), + "max": round(max(v), digits)} + + +def cer(ref: str, hyp: str) -> float: + """Character error rate after removing spaces and punctuation.""" + def norm(s: str) -> str: + return re.sub(r"[\W_]+", "", s, flags=re.UNICODE).lower() + r, h = norm(ref), norm(hyp) + if not r: + return 0.0 if not h else 1.0 + prev = list(range(len(h) + 1)) + for i, rc in enumerate(r, 1): + cur = [i] + [0] * len(h) + for j, hc in enumerate(h, 1): + cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (rc != hc)) + prev = cur + return round(prev[-1] / len(r), 4) + + +def stage_args(argv: list[str] | None = None) -> tuple[Path, Path, dict[str, str]]: + """Common CLI for stage scripts: --out FILE --work DIR [--opt k=v ...].""" + import argparse + p = argparse.ArgumentParser() + p.add_argument("--out", required=True) + p.add_argument("--work", required=True) + p.add_argument("--opt", action="append", default=[]) + a = p.parse_args(argv) + opts = dict(o.split("=", 1) for o in a.opt if "=" in o) + return Path(a.out), Path(a.work), opts diff --git a/scripts/m0/m0lib/envs.py b/scripts/m0/m0lib/envs.py new file mode 100644 index 0000000..881f091 --- /dev/null +++ b/scripts/m0/m0lib/envs.py @@ -0,0 +1,136 @@ +"""Per-stage Python environments, created with uv inside the work folder. + +Each heavy stage gets its own venv because their pins conflict (MeloTTS needs +transformers 4.27 on Python 3.11, CosyVoice needs transformers 4.51, ...). +""" + +from __future__ import annotations + +import io +import os +import shutil +import sys +import urllib.request +import zipfile +from pathlib import Path + +from m0lib.common import IS_WINDOWS, run_logged, say + +CU128_INDEX = "https://download.pytorch.org/whl/cu128" +TORCH_CU128 = ["torch==2.11.0", "torchaudio==2.11.0", "--index-url", CU128_INDEX] +# After the CUDA torch is in, later steps must not swap it for another build +# (openai-whisper's Linux triton<3 pin silently downgraded torch to 2.3.1). +TORCH_PIN = "torch==2.11.0+cu128\ntorchaudio==2.11.0+cu128\n" +MELO_URL = "https://github.com/myshell-ai/MeloTTS/archive/209145371cff8fc3bd60d7be902ea69cbdb7965a.tar.gz" +COSY_ZIP = "https://github.com/FunAudioLLM/CosyVoice/archive/074ca6dc9e80a2f424f1f74b48bdd7d3fea531cc.zip" +MATCHA_ZIP = "https://github.com/shivammehta25/Matcha-TTS/archive/dd9105b34bf2be2230f4aa1e4769fb586a3c824e.zip" + +UNINSTALL = "--m0-uninstall" + +# name -> (python version, list of install steps; each step is the argument list for `uv pip install`) +SPECS: dict[str, tuple[str, list[list[str]]]] = { + "base": ("3.12", [["pillow==12.2.0"]]), + "stt": ("3.12", [["faster-whisper==1.2.1", "ctranslate2==4.8.1", "nvidia-cublas-cu12==12.9.2.10", + "nvidia-cudnn-cu12==9.24.0.43"]]), + # MeloTTS pulls mecab-python3 (package dir "MeCab", Japanese only) and the + # Korean g2p needs python-mecab-ko (package dir "mecab"). On Windows the two + # folder names are the same folder, so mecab-python3 is removed first and + # tts_melo.py stubs the Japanese tagger. + "melo": ("3.11", [TORCH_CU128, + [MELO_URL], + [UNINSTALL, "mecab-python3"], + ["python-mecab-ko==1.3.7", "python-mecab-ko-dic==2.1.1.post2"]]), + "cosy": ("3.10", [TORCH_CU128, + # openai-whisper is an sdist whose setup.py imports pkg_resources. + ["setuptools<81", "wheel"], + ["--no-build-isolation", "--no-deps", "openai-whisper==20231117"], + ["numba", "tiktoken", "more-itertools", "tqdm"], + ["conformer==0.3.2", "diffusers==0.29.0", "HyperPyYAML==1.2.3", "inflect==7.3.1", + "librosa==0.10.2", "modelscope==1.20.0", "numpy==1.26.4", "omegaconf==2.3.0", + "onnxruntime==1.18.0", "soundfile==0.12.1", + "transformers==4.51.3", "x-transformers==2.11.24", "wetext==0.0.4", "hydra-core==1.3.2", + "lightning==2.2.4", "matplotlib==3.7.5", "huggingface_hub==0.36.2", "rich==13.7.1", + "pyarrow==18.1.0", "gdown==5.1.0", "wget==3.2", "pyworld==0.3.4"]]), +} + + +def venv_python(venv: Path) -> Path: + return venv / ("Scripts/python.exe" if IS_WINDOWS else "bin/python") + + +def find_uv() -> str: + uv = os.environ.get("M0_UV") or shutil.which("uv") + if not uv: + raise RuntimeError("uv not found (run_m0.ps1 installs it)") + return uv + + +def ensure_env(name: str, work: Path, log: Path) -> Path: + """Create the venv for ``name`` once; returns its python. Raises on failure.""" + py_ver, steps = SPECS[name] + venv = work / "venvs" / name + py = venv_python(venv) + done = venv / ".m0-ready" + if done.exists() and py.exists(): + return py + uv = find_uv() + say(f" [{name}] Python {py_ver} 가상환경 준비 중 (처음 한 번만, 몇 분 걸립니다)") + if run_logged([uv, "venv", "--clear", "--python", py_ver, str(venv)], log, timeout=1800) != 0: + raise RuntimeError(f"uv venv failed for {name} (see {log.name})") + pinned = False + for i, args in enumerate(steps, 1): + say(f" [{name}] 패키지 설치 {i}/{len(steps)}") + extra: list[str] = [] + if pinned: + pin = venv / "torch-pin.txt" + pin.write_text(TORCH_PIN, encoding="utf-8") + extra = ["-c", str(pin), "--extra-index-url", CU128_INDEX, "--index-strategy", "unsafe-best-match"] + if args[0] == UNINSTALL: + run_logged([uv, "pip", "uninstall", "--python", str(py), *args[1:]], log, timeout=600) + _remove_package_dirs(venv, {a.lower() for a in args[1:]}) + continue + if run_logged([uv, "pip", "install", "--python", str(py), *args, *extra], log, timeout=3600) != 0: + raise RuntimeError(f"package install step {i} failed for {name} (see {log.name})") + pinned = pinned or args is TORCH_CU128 + if name == "melo": + # The full `unidic` package ships without its dictionary; with it removed + # mecab-python3 falls back to unidic_lite (same fix as docker/setup-melo.sh). + run_logged([uv, "pip", "uninstall", "--python", str(py), "unidic"], log, timeout=600) + done.write_text("ok", encoding="utf-8") + return py + + +def _remove_package_dirs(venv: Path, dists: set[str]) -> None: + """Remove folders an uninstall can leave behind (a case-insensitive clash on Windows).""" + names = {"mecab-python3": "mecab"} + targets = {names.get(d, d) for d in dists} + for sp in list(venv.glob("Lib/site-packages")) + list(venv.glob("lib/python*/site-packages")): + for entry in sp.iterdir(): + if entry.is_dir() and entry.name.lower() in targets: + shutil.rmtree(entry, ignore_errors=True) + + +def _unzip_github(url: str, dest: Path, log: Path) -> None: + say(f" 소스 받는 중: {url.split('/')[4]}") + with urllib.request.urlopen(url, timeout=600) as resp: + data = resp.read() + with zipfile.ZipFile(io.BytesIO(data)) as z: + top = z.namelist()[0].split("/")[0] + tmp = dest.parent / (dest.name + ".tmp") + shutil.rmtree(tmp, ignore_errors=True) + z.extractall(tmp) + shutil.rmtree(dest, ignore_errors=True) + (tmp / top).rename(dest) + shutil.rmtree(tmp, ignore_errors=True) + with open(log, "a", encoding="utf-8") as f: + f.write(f"unpacked {url} -> {dest.name}\n") + + +def ensure_cosy_source(work: Path, log: Path) -> Path: + src = work / "src" / "CosyVoice" + if (src / "cosyvoice").is_dir() and (src / "third_party" / "Matcha-TTS" / "matcha").is_dir(): + return src + src.parent.mkdir(parents=True, exist_ok=True) + _unzip_github(COSY_ZIP, src, log) + _unzip_github(MATCHA_ZIP, src / "third_party" / "Matcha-TTS", log) + return src diff --git a/scripts/m0/m0lib/llm.py b/scripts/m0/m0lib/llm.py new file mode 100644 index 0000000..0fb4713 --- /dev/null +++ b/scripts/m0/m0lib/llm.py @@ -0,0 +1,245 @@ +"""Stage: LLM through Strata (install, start, measure, restart timings). + +Options (--opt key=value): + model=fast|recommended which preset to measure (default fast) + url=http://.../v1 measure an already running server instead (no Strata management) + context=32768 context size to start Strata with +""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from m0lib.common import IS_WINDOWS, GpuSampler, StageResult, run_logged, say, scrub, stage_args # noqa: E402 +from m0lib.llm_bench import bench_single, test_decode, test_images, test_parallel, test_vram_shrink, warmup # noqa: E402 +from m0lib.llm_client import ChatClient, http_json # noqa: E402 + +PRESETS = { + "fast": {"family": "qwen36", "model": "ud-q4_k_xl", "label": "Qwen3.6-35B-A3B UD-Q4_K_XL"}, + "recommended": {"family": "qwen", "model": "q2_0", "label": "Qwen3.8-Flash-Next Q2_0 (ISTA-DASLab)"}, +} +PORT = 8080 +INSTALL_URL = "https://github.com/Niko1221/Strata/releases/latest/download/install.ps1" + + +class Strata: + def __init__(self, work: Path, log: Path): + self.home = work / "strata" + self.models = work / "models" / "llm" + self.log = log + # Keep Strata's engine, state and models inside the work folder: the + # installer honours STRATA_DATA_DIR, the engine components + # STRATA_COMPONENTS_DIR, and the per-user state follows LOCALAPPDATA. + self.env = dict(os.environ, STRATA_DATA_DIR=str(self.home), STRATA_NO_PATH="1", + STRATA_HOME=str(self.home / "home"), STRATA_COMPONENTS_DIR=str(self.home / "components"), + PYTHONUTF8="1") + if IS_WINDOWS: + self.env["LOCALAPPDATA"] = str(self.home / "localappdata") + (self.home / "localappdata").mkdir(parents=True, exist_ok=True) + self._flags: set[str] | None = None + + @property + def exe(self) -> Path: + return self.home / "terminal" / "current" / "bin" / ("strata.cmd" if IS_WINDOWS else "strata") + + def install(self) -> bool: + if self.exe.exists(): + return True + self.home.mkdir(parents=True, exist_ok=True) + say(" Strata 터미널 설치 중 (작업 폴더 안에만 설치하고 PATH는 바꾸지 않습니다)") + if IS_WINDOWS: + cmd = ["powershell", "-NoProfile", "-ExecutionPolicy", "Bypass", "-Command", f"irm {INSTALL_URL} | iex"] + else: + cmd = ["sh", "-c", f"curl -fsSL {INSTALL_URL[:-3]}sh -o install.sh && sh install.sh"] + rc = run_logged(cmd, self.log, timeout=1800, env=self.env, cwd=str(self.home), echo=True) + return rc == 0 and self.exe.exists() + + def cmd(self, *args: str) -> list[str]: + return [str(self.exe), *args] + + def supported_flags(self) -> set[str]: + if self._flags is None: + p = subprocess.run(self.cmd("launch", "--help"), capture_output=True, text=True, env=self.env, + encoding="utf-8", errors="replace", timeout=120) + with open(self.log, "a", encoding="utf-8") as f: + f.write(scrub(p.stdout + p.stderr)) + self._flags = set(re.findall(r"--[a-z0-9][a-z0-9-]*", p.stdout + p.stderr)) + return self._flags + + def launch_args(self, preset: dict, context: int, parallel: int, vision: str, extra_sets: list[str], + restart: bool) -> tuple[list[str], list[str]]: + want = [("--family", preset["family"]), ("--model", preset["model"]), ("--data-dir", str(self.models)), + ("--context", str(context)), ("--vision", vision), ("--parallel", str(parallel)), + ("--access", "local"), ("--port", str(PORT)), ("--no-api-key", None), ("--yes", None), + ("--no-tui", None)] + if restart: + want.append(("--restart", None)) + for s in extra_sets: + want.append(("--set", s)) + flags = self.supported_flags() + args, dropped = [], [] + for k, v in want: + if flags and k not in flags: + dropped.append(k) + continue + args += [k] + ([v] if v is not None else []) + return args, dropped + + def plan(self, preset: dict, context: int) -> object: + args, _ = self.launch_args(preset, context, 1, "on", [], False) + args = [a for a in args if a not in ("--yes", "--no-tui")] + ["--plan", "--output", "json"] + p = subprocess.run(self.cmd("launch", *args), capture_output=True, text=True, env=self.env, + encoding="utf-8", errors="replace", timeout=600) + with open(self.log, "a", encoding="utf-8") as f: + f.write(scrub(p.stdout + p.stderr)) + try: + return json.loads(p.stdout) + except ValueError: + return {"exit": p.returncode, "text": scrub(p.stdout[-3000:] + p.stderr[-2000:])} + + def launch(self, preset: dict, context: int, parallel: int, vision: str = "on", + extra_sets: list[str] | None = None, restart: bool = False, timeout: float = 4 * 3600) -> dict: + args, dropped = self.launch_args(preset, context, parallel, vision, extra_sets or [], restart) + last = [0.0] + + def progress(line: str) -> bool: + # Downloads print many progress lines; show one every 15 s. + if "%" in line: + if time.monotonic() - last[0] < 15: + return False + last[0] = time.monotonic() + return True + + t0 = time.perf_counter() + rc = run_logged(self.cmd("launch", *args), self.log, timeout=timeout, env=self.env, echo=True, + echo_filter=progress) + t_cmd = time.perf_counter() - t0 + ready = wait_ready(f"http://127.0.0.1:{PORT}", timeout=1800 if rc == 0 else 30) + return {"args": scrub(" ".join(args)), "dropped_flags": dropped, "exit": rc, + "command_seconds": round(t_cmd, 1), "ready": ready is not None, + "ready_seconds": round(time.perf_counter() - t0, 1) if ready is not None else None} + + def stop(self) -> None: + run_logged(self.cmd("launch", "--stop"), self.log, timeout=300, env=self.env) + time.sleep(3) + + def status(self) -> object: + p = subprocess.run(self.cmd("launch", "--status", "--output", "json"), capture_output=True, text=True, + env=self.env, encoding="utf-8", errors="replace", timeout=120) + try: + return json.loads(p.stdout) + except ValueError: + return scrub(p.stdout[-2000:]) + + +def wait_ready(root: str, timeout: float) -> float | None: + t0 = time.perf_counter() + while time.perf_counter() - t0 < timeout: + try: + code, body = http_json("GET", root + "/v1/models", timeout=5) + if code == 200 and isinstance(body, dict) and body.get("data"): + return time.perf_counter() - t0 + except Exception: + pass + time.sleep(2) + return None + + +def server_info(base: str) -> dict: + info = {} + for path in ("/models", "/status"): + try: + code, body = http_json("GET", base + path, timeout=10) + info[path] = body if code == 200 else {"http": code} + except Exception as e: + info[path] = {"error": str(e)} + return info + + +def measure_external(r: StageResult, url: str, context: int) -> None: + c = ChatClient(url) + r["server"] = {"url": url, "model": c.model, "info": server_info(c.base)} + with GpuSampler() as g: + bench_single(r, c, context) + test_parallel(r, c) + r["gpu"] = g.summary() + + +def measure_strata(r: StageResult, work: Path, which: str, context: int) -> None: + preset = PRESETS[which] + r["preset"] = dict(preset, key=which, context=context) + s = Strata(work, work / "logs" / f"strata_{which}.log") + if not s.install(): + r.error("install", "Strata 설치 실패 (logs/strata_*.log 참고)") + return + r["plan"] = s.plan(preset, context) + r.save() + + # 1) Start (the first start downloads the model and engine). + say(f" {preset['label']} 시작 중. 처음에는 모델을 내려받으므로 오래 걸립니다.") + s.stop() + with GpuSampler(interval=2) as g: + r["launch_cold"] = s.launch(preset, context, parallel=1) + r["launch_cold"].update(g.summary()) + r.save() + if not r["launch_cold"]["ready"]: + r.error("launch", "서버가 준비되지 않았습니다") + r["status_after_fail"] = s.status() + return + base = f"http://127.0.0.1:{PORT}/v1" + c = ChatClient(base) + r["server"] = {"model": c.model, "info": server_info(base), "status": s.status()} + with GpuSampler() as g: + bench_single(r, c, context) + r["gpu_single"] = g.summary() + r.save() + + # 2) Restart with the files already in RAM/page cache: what a game-mode switch costs. + # Also: two slots, the elastic VRAM cache and the GPU picture reader. + say(" 다시 시작 시간 측정 (동시 2개, VRAM 조절 켜기, 이미지 GPU 처리)") + r["launch_restart"] = s.launch(preset, context, parallel=2, vision="gpu", extra_sets=["vram-elastic"], + restart=True, timeout=3600) + r.save() + if r["launch_restart"]["ready"]: + c = ChatClient(base) + r["server_restart"] = {"status": s.status()} + for fn in (lambda: warmup(r, c), lambda: test_decode(r, c, key="decode_parallel2"), lambda: test_parallel(r, c), + lambda: test_images(r, c, key="images_gpu_reader"), lambda: test_vram_shrink(r, c)): + try: + fn() + except Exception as e: + r.error("restart tests", e) + r.save() + + # 3) Game mode (b): switching from the recommended model to the fast one. + if which == "recommended" and (work / "results" / "llm_fast.json").exists(): + say(" 게임 모드 (b): 권장 모델 -> 빠름 모델 전환 시간") + r["switch_to_fast"] = s.launch(PRESETS["fast"], context, parallel=1, restart=True, timeout=3600) + s.stop() + + +def main() -> int: + out, work, opts = stage_args() + which = opts.get("model", "fast") + context = int(opts.get("context", "32768")) + r = StageResult(f"llm_{which}", out) + try: + if opts.get("url"): + measure_external(r, opts["url"], context) + else: + measure_strata(r, work, which, context) + except Exception as e: + r.error("llm", e) + r.save("ok" if not r["errors"] else "partial") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/m0/m0lib/llm_bench.py b/scripts/m0/m0lib/llm_bench.py new file mode 100644 index 0000000..4d70940 --- /dev/null +++ b/scripts/m0/m0lib/llm_bench.py @@ -0,0 +1,382 @@ +"""LLM measurements against an OpenAI-compatible server (Strata or another). + +Every test is fail-open: an error is recorded in the result and the next test +runs. The answers are kept so Korean quality can be read afterwards. +""" + +from __future__ import annotations + +import base64 +import io +import json +import os +import random +import threading +import time +from pathlib import Path +from typing import Any + +from m0lib.common import StageResult, gpu_memory_mib, say, stats +from m0lib.llm_client import ChatClient, http_json + +NO_THINK = {"chat_template_kwargs": {"enable_thinking": False}} + +VOICE_SYSTEM = ("너는 디스코드 음성 채널에서 대답하는 비서 '자비스'다. 대답은 소리 내어 읽히므로 " + "한국어 존댓말로 짧게, 두 문장 이내로 말한다. 목록, 마크다운, 이모지는 쓰지 않는다.") + +# -------------------------------------------------------------------------- +# Context length / TTFT + +_WORDS = ("오늘 내일 회의 일정 날씨 음악 게임 친구 저녁 메뉴 영화 주말 기록 화면 모니터 채널 " + "서버 사진 여행 공부 운동 커피 버스 지하철 약속 생일 선물 노래 방송 뉴스 시장 가격 " + "연습 준비 정리 확인 계획 결과 문제 해결 방법 이유 생각 이야기 질문 대답 시간 장소").split() + + +def filler(n_chars: int, seed: int) -> str: + rnd = random.Random(seed) + parts, size = [], 0 + while size < n_chars: + s = " ".join(rnd.choice(_WORDS) for _ in range(rnd.randint(6, 12))) + "에 대해 이야기했다. " + parts.append(s) + size += len(s) + return "".join(parts) + + +def test_context(r: StageResult, c: ChatClient, ctx_limit: int) -> None: + say(" - 맥락 길이별 첫 토큰 시간") + rows = [] + # Characters per token differ between tokenizers: start with a guess and + # correct it from the server's prompt_tokens after the first request. + chars_per_token = 1.5 + for target in (256, 2048, 8192, 16384, 24576): + if target > ctx_limit * 0.8: + continue + seed = random.randint(0, 10**9) + doc = f"[문서 {seed}]\n" + filler(int(target * chars_per_token), seed) + msgs = [{"role": "system", "content": VOICE_SYSTEM}, + {"role": "user", "content": doc + "\n\n위 글의 주제를 한 문장으로 말해 줘."}] + row: dict[str, Any] = {"target_tokens": target} + try: + a = c.chat(msgs, max_tokens=48, extra=NO_THINK) + row.update(prompt_tokens=(a["usage"] or {}).get("prompt_tokens"), ttft=a["ttft"], + first_sentence=a["first_sentence"], total=a["total"]) + if row["prompt_tokens"]: + chars_per_token = len(doc) / row["prompt_tokens"] + # Same long prefix, a new question at the end: shows prompt-cache reuse. + msgs2 = msgs[:1] + [{"role": "user", "content": doc + "\n\n위 글에서 가장 많이 나온 낱말 하나만 말해 줘."}] + b = c.chat(msgs2, max_tokens=24, extra=NO_THINK) + row.update(ttft_same_prefix=b["ttft"]) + say(f" {row.get('prompt_tokens')} 토큰: 첫 토큰 {row['ttft']}s, 같은 앞부분 재사용 {row['ttft_same_prefix']}s") + except Exception as e: + r.error(f"context {target}", e) + rows.append(row) + r["context_ttft"] = rows + + +def test_decode(r: StageResult, c: ChatClient, key: str = "decode") -> None: + say(" - 생성 속도 (tok/s)") + runs = [] + for i in range(3): + try: + a = c.chat([{"role": "user", "content": f"({i}) 한국의 사계절을 각각 세 문장씩 자세히 설명해 줘."}], + max_tokens=320, extra=NO_THINK) + runs.append({k: a.get(k) for k in ("ttft", "total", "decode_tok_s", "usage", "server_timings")}) + except Exception as e: + r.error(f"{key} {i}", e) + r[key] = {"runs": runs, "decode_tok_s": stats(x.get("decode_tok_s") for x in runs), + "ttft": stats(x.get("ttft") for x in runs)} + say(f" 중앙값 {r[key]['decode_tok_s'].get('median')} tok/s") + try: + code, body = http_json("GET", c.base + "/status", key=c.key, timeout=10) + if code == 200 and isinstance(body, dict): + r[f"{key}_server_last_timings"] = body.get("last_timings") + except Exception: + pass + + +def test_thinking_default(r: StageResult, c: ChatClient) -> None: + """Does the model think by default, and what does that cost the first answer token?""" + try: + a = c.chat([{"role": "user", "content": "자비스, 오늘 저녁 메뉴 하나만 추천해 줘."}], max_tokens=400) + r["thinking_default"] = {k: a.get(k) for k in ("ttft", "first_reasoning", "reasoning_chars", "total", "text")} + except Exception as e: + r.error("thinking_default", e) + + +# -------------------------------------------------------------------------- +# Tool calling + +TOOLS = [ + {"type": "function", "function": { + "name": "get_weather", "description": "지정한 도시의 현재 날씨를 가져온다.", + "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "도시 이름"}}, + "required": ["city"]}}}, + {"type": "function", "function": { + "name": "set_timer", "description": "몇 분 뒤에 알리는 타이머를 맞춘다.", + "parameters": {"type": "object", "properties": {"minutes": {"type": "integer"}}, "required": ["minutes"]}}}, + {"type": "function", "function": { + "name": "view_screen", "description": "사용자 PC의 화면을 캡처해서 본다.", + "parameters": {"type": "object", "properties": { + "monitor": {"type": "integer", "description": "모니터 번호 (1부터)"}, + "quality": {"type": "string", "enum": ["fast", "normal", "detail"]}}, "required": ["monitor"]}}}, +] + +TOOL_CASES = [ + ("서울 날씨 어때?", "get_weather", lambda a: "서울" in str(a.get("city", "")) or "Seoul" in str(a.get("city", ""))), + ("5분 타이머 맞춰 줘.", "set_timer", lambda a: int(a.get("minutes", 0)) == 5), + ("두 번째 모니터에 뭐 떠 있어?", "view_screen", lambda a: int(a.get("monitor", 0)) == 2), + ("안녕, 오늘 기분 어때?", None, None), +] + + +def test_tools(r: StageResult, c: ChatClient) -> None: + say(" - 도구 호출") + rows = [] + for prompt, want, check in TOOL_CASES: + for i in range(3): + row: dict[str, Any] = {"prompt": prompt, "expected": want, "try": i + 1} + try: + a = c.chat([{"role": "system", "content": VOICE_SYSTEM}, {"role": "user", "content": prompt}], + max_tokens=200, tools=TOOLS, extra=NO_THINK) + calls = a["tool_calls"] + row.update(total=a["total"], first_tool=a["first_tool"], ttft=a["ttft"], finish=a["finish"], + calls=calls, text=a["text"][:200]) + if want is None: + row["ok"] = not calls + else: + ok = bool(calls) and calls[0]["name"] == want + if ok: + try: + ok = bool(check(json.loads(calls[0]["arguments"] or "{}"))) + except Exception: + ok = False + row["ok"] = ok + except Exception as e: + row["error"] = str(e)[:300] + row["ok"] = False + rows.append(row) + ok = sum(1 for x in rows if x.get("ok")) + r["tools"] = {"cases": rows, "ok": ok, "total": len(rows)} + say(f" 성공 {ok}/{len(rows)}") + # Round trip: give the tool result back and read the final answer. + try: + msgs = [{"role": "system", "content": VOICE_SYSTEM}, {"role": "user", "content": "서울 날씨 어때?"}] + a = c.chat(msgs, max_tokens=200, tools=TOOLS, extra=NO_THINK) + if a["tool_calls"]: + call = a["tool_calls"][0] + msgs += [{"role": "assistant", "content": "", "tool_calls": [ + {"id": "call_1", "type": "function", + "function": {"name": call["name"], "arguments": call["arguments"]}}]}, + {"role": "tool", "tool_call_id": "call_1", + "content": json.dumps({"city": "서울", "temp_c": 13, "sky": "맑음"}, ensure_ascii=False)}] + b = c.chat(msgs, max_tokens=120, tools=TOOLS, extra=NO_THINK) + r["tools_round_trip"] = {"ttft": b["ttft"], "first_sentence": b["first_sentence"], "text": b["text"], + "uses_result": "13" in b["text"]} + except Exception as e: + r.error("tools round trip", e) + + +# -------------------------------------------------------------------------- +# Pictures + +CODE_BIG = "4827" +CODE_SMALL = "QX-7316" + + +def _font(size: int): + from PIL import ImageFont + for p in (r"C:\Windows\Fonts\malgun.ttf", r"C:\Windows\Fonts\malgunbd.ttf", + "/usr/share/fonts/truetype/nanum/NanumGothic.ttf", + "/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc"): + if os.path.exists(p): + return ImageFont.truetype(p, size) + return ImageFont.load_default(size=size) + + +def make_screen(seed: int = 0) -> "Any": + """A synthetic 4K 'desktop' with a big number, a small code and Korean text. + + ``seed`` changes a few pixels' worth of content so a server cannot answer a + repeated try from its picture cache. + """ + from PIL import Image, ImageDraw + img = Image.new("RGB", (3840, 2160), (32, 56, 92)) + d = ImageDraw.Draw(img) + d.rectangle([200, 160, 2400, 1500], fill=(245, 245, 245)) + d.rectangle([200, 160, 2400, 230], fill=(220, 220, 220)) + d.text((230, 172), "메모장 - 오늘 할 일.txt", font=_font(40), fill=(20, 20, 20)) + lines = ["1. 오후 세 시 회의 자료 정리", "2. 저녁 장보기: 두부, 계란, 대파", "3. 디스코드 서버 공지 올리기"] + for i, line in enumerate(lines): + d.text((260, 300 + i * 90), line, font=_font(56), fill=(30, 30, 30)) + d.rectangle([2700, 300, 3500, 800], outline=(220, 30, 30), width=14) + d.text((2830, 420), CODE_BIG, font=_font(220), fill=(220, 30, 30)) + d.text((260, 1400), f"확인 코드: {CODE_SMALL}", font=_font(26), fill=(90, 90, 90)) + d.rectangle([0, 2080, 3840, 2160], fill=(20, 20, 20)) + d.text((3300, 2095), f"오후 {seed % 12 + 1}:{seed % 60:02d}", font=_font(44), fill=(230, 230, 230)) + return img + + +def _jpeg_b64(img: Any, long_side: int) -> tuple[str, float, tuple[int, int]]: + t0 = time.perf_counter() + w, h = img.size + s = long_side / max(w, h) + im = img.resize((max(1, round(w * s)), max(1, round(h * s)))) if s < 1 else img + buf = io.BytesIO() + im.save(buf, "JPEG", quality=85) + return base64.b64encode(buf.getvalue()).decode(), time.perf_counter() - t0, im.size + + +def picture_parts(img: Any, tier: str) -> tuple[list[dict], float, list]: + if tier == "detail": + w, h = img.size + tiles = [img.crop((x, y, x + w // 2, y + h // 2)) for y in (0, h // 2) for x in (0, w // 2)] + enc = [_jpeg_b64(t, 1280) for t in tiles] + else: + enc = [_jpeg_b64(img, {"fast": 768, "normal": 1280}[tier])] + parts = [{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b}"}} for b, _, _ in enc] + return parts, sum(t for _, t, _ in enc), [s for _, _, s in enc] + + +def test_images(r: StageResult, c: ChatClient, key: str = "images") -> None: + say(" - 이미지 읽기 (화질 단계별)") + rows = [] + question = ("이 화면에서 빨간 상자 안의 숫자와, 작은 글씨로 적힌 '확인 코드'를 읽어 줘. " + "모르면 모른다고 해. 형식: 숫자=..., 코드=...") + seed = random.randint(0, 10**6) + for tier in ("fast", "normal", "detail"): + for i in range(3): + seed += 1 + # The first picture after a start may include starting the picture reader. + row: dict[str, Any] = {"tier": tier, "try": i + 1} + try: + parts, prep, sizes = picture_parts(make_screen(seed), tier) + msgs = [{"role": "user", "content": parts + [{"type": "text", "text": question}]}] + a = c.chat(msgs, max_tokens=60, extra=NO_THINK) + row.update(prep_s=round(prep, 3), sizes=sizes, ttft=a["ttft"], total=a["total"], + prompt_tokens=(a["usage"] or {}).get("prompt_tokens"), text=a["text"][:200], + big_ok=CODE_BIG in a["text"], small_ok=CODE_SMALL.replace("-", "") in + a["text"].replace("-", "").replace(" ", "").upper()) + say(f" {tier}: 첫 토큰 {row['ttft']}s, 큰 숫자 {'O' if row['big_ok'] else 'X'}, " + f"작은 코드 {'O' if row['small_ok'] else 'X'}") + except Exception as e: + row["error"] = str(e)[:300] + say(f" {tier}: 실패 ({row['error'][:120]})") + rows.append(row) + r[key] = rows + r[key + "_summary"] = {t: {"ttft_after_first": stats(x.get("ttft") for x in rows if x["tier"] == t and x["try"] > 1), + "big_ok": sum(1 for x in rows if x["tier"] == t and x.get("big_ok")), + "small_ok": sum(1 for x in rows if x["tier"] == t and x.get("small_ok")), + "tries": sum(1 for x in rows if x["tier"] == t)} + for t in ("fast", "normal", "detail")} + + +# -------------------------------------------------------------------------- +# Korean quality: answers are saved for reading + +KOREAN_PROMPTS = [ + ("voice", "자비스, 오늘 비 온다는데 우산 챙겨야 할까?"), + ("voice", "자비스, 라면 맛있게 끓이는 방법 알려 줘."), + ("voice", "자비스, 아까 내가 무슨 노래 틀어 달라고 했지? (기억이 없으면 솔직하게 말해 줘)"), + ("voice", "자비스, 3 곱하기 17 더하기 9는 얼마야?"), + ("plain", "존댓말과 반말의 차이를 초등학생에게 설명하듯 세 문장으로 말해 줘."), + ("plain", "다음 문장을 자연스러운 한국어로 고쳐 줘: '나는 어제 친구를 만나는 것을 했다 그리고 영화를 보았다.'"), + ("plain", "\"내일 오후 2시에 강남역에서 민수랑 만나기로 했어\"에서 시간, 장소, 사람을 JSON으로 뽑아 줘."), + ("plain", "Translate to natural Korean: 'I'll be right back, just grabbing a coffee.'"), +] + + +def test_korean(r: StageResult, c: ChatClient) -> None: + say(" - 한국어 답변 모음") + rows = [] + for kind, prompt in KOREAN_PROMPTS: + msgs = ([{"role": "system", "content": VOICE_SYSTEM}] if kind == "voice" else []) + \ + [{"role": "user", "content": prompt}] + try: + a = c.chat(msgs, max_tokens=300, extra=NO_THINK) + rows.append({"kind": kind, "prompt": prompt, "answer": a["text"], "ttft": a["ttft"], + "first_sentence": a["first_sentence"], "total": a["total"]}) + except Exception as e: + rows.append({"kind": kind, "prompt": prompt, "error": str(e)[:300]}) + r["korean"] = rows + r["voice_first_sentence"] = stats(x.get("first_sentence") for x in rows if x["kind"] == "voice") + + +# -------------------------------------------------------------------------- +# Several requests at once + +def test_parallel(r: StageResult, c: ChatClient, n_list=(2, 4)) -> None: + say(" - 동시 요청") + out = {} + for n in n_list: + results: list[dict] = [{} for _ in range(n)] + + def one(i: int) -> None: + try: + a = c.chat([{"role": "user", "content": f"({i}) 좋아하는 계절 하나를 골라 이유를 다섯 문장으로 말해 줘."}], + max_tokens=160, extra=NO_THINK) + results[i] = {k: a.get(k) for k in ("ttft", "total", "decode_tok_s")} + results[i]["completion_tokens"] = (a.get("usage") or {}).get("completion_tokens") + except Exception as e: + results[i] = {"error": str(e)[:200]} + + t0 = time.perf_counter() + th = [threading.Thread(target=one, args=(i,)) for i in range(n)] + for t in th: + t.start() + for t in th: + t.join() + wall = time.perf_counter() - t0 + toks = sum(x.get("completion_tokens") or 0 for x in results) + out[str(n)] = {"wall_s": round(wall, 2), "sum_tokens": toks, "aggregate_tok_s": round(toks / wall, 1), + "ttft": stats(x.get("ttft") for x in results), "requests": results} + say(f" {n}개 동시: 전체 {out[str(n)]['aggregate_tok_s']} tok/s, " + f"첫 토큰 최대 {out[str(n)]['ttft'].get('max')}s") + r["parallel"] = out + + +# -------------------------------------------------------------------------- +# Game mode (a): give VRAM back with POST /v1/vram and take it again + +def test_vram_shrink(r: StageResult, c: ChatClient, reserve_mib: int = 8000) -> None: + say(f" - 게임 모드 (a): VRAM {reserve_mib} MiB 비우기") + out: dict[str, Any] = {"reserve_mib": reserve_mib, "gpu_before": gpu_memory_mib()} + root = c.base[:-3] if c.base.endswith("/v1") else c.base + try: + t0 = time.perf_counter() + code, body = http_json("POST", root + "/v1/vram", {"reserve_mib": reserve_mib}, timeout=400, key=c.key) + out["shrink"] = {"http": code, "seconds": round(time.perf_counter() - t0, 3), "body": body} + out["gpu_after_shrink"] = gpu_memory_mib() + if code == 200: + a = c.chat([{"role": "user", "content": "한국의 사계절을 각각 세 문장씩 설명해 줘."}], + max_tokens=200, extra=NO_THINK) + out["decode_tok_s_shrunk"] = a.get("decode_tok_s") + out["ttft_shrunk"] = a.get("ttft") + t0 = time.perf_counter() + code2, body2 = http_json("POST", root + "/v1/vram", {"reserve_mib": None}, timeout=400, key=c.key) + out["grow"] = {"http": code2, "seconds": round(time.perf_counter() - t0, 3), "body": body2} + out["gpu_after_grow"] = gpu_memory_mib() + say(f" 비우기 HTTP {code} ({out['shrink']['seconds']}s), 되돌리기 HTTP {code2} ({out['grow']['seconds']}s), " + f"비운 상태 속도 {out.get('decode_tok_s_shrunk')} tok/s") + except Exception as e: + r.error("vram shrink", e) + r["vram_shrink"] = out + + +def warmup(r: StageResult, c: ChatClient) -> None: + """One throwaway request so model loading is not counted as answer time.""" + try: + a = c.chat([{"role": "user", "content": "안녕"}], max_tokens=8, extra=NO_THINK) + r["warmup_total_s"] = a["total"] + except Exception as e: + r.error("warmup", e) + + +def bench_single(r: StageResult, c: ChatClient, ctx_limit: int) -> None: + """The tests that need one request at a time.""" + for fn in (lambda: warmup(r, c), lambda: test_decode(r, c), lambda: test_thinking_default(r, c), lambda: test_context(r, c, ctx_limit), + lambda: test_tools(r, c), lambda: test_images(r, c), lambda: test_korean(r, c)): + try: + fn() + except Exception as e: + r.error("bench", e) + r.save() diff --git a/scripts/m0/m0lib/llm_client.py b/scripts/m0/m0lib/llm_client.py new file mode 100644 index 0000000..6cdb411 --- /dev/null +++ b/scripts/m0/m0lib/llm_client.py @@ -0,0 +1,146 @@ +"""Minimal OpenAI-compatible chat client with timing (standard library only).""" + +from __future__ import annotations + +import json +import re +import time +import urllib.error +import urllib.request +from typing import Any + +# A sentence ends at . ? ! or a CJK full stop, followed by space/end, or at a newline. +_SENTENCE_END = re.compile(r"([.?!。?!][\"')\]]*(\s|$))|\n") + + +def http_json(method: str, url: str, body: Any = None, timeout: float = 30, key: str | None = None) -> tuple[int, Any]: + data = json.dumps(body).encode("utf-8") if body is not None else None + req = urllib.request.Request(url, data=data, method=method) + req.add_header("Content-Type", "application/json") + if key: + req.add_header("Authorization", f"Bearer {key}") + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + raw = resp.read().decode("utf-8", "replace") + return resp.status, (json.loads(raw) if raw.strip() else None) + except urllib.error.HTTPError as e: + raw = e.read().decode("utf-8", "replace") + try: + return e.code, json.loads(raw) + except ValueError: + return e.code, raw[:2000] + + +class ChatClient: + def __init__(self, base_url: str, model: str | None = None, key: str | None = None, + extra: dict[str, Any] | None = None, timeout: float = 600): + self.base = base_url.rstrip("/") + self.key = key + self.timeout = timeout + self.extra = extra or {} + self.model = model or self.first_model() + + def first_model(self) -> str: + code, body = http_json("GET", f"{self.base}/models", key=self.key) + if code != 200 or not isinstance(body, dict) or not body.get("data"): + raise RuntimeError(f"/models answered {code}: {str(body)[:300]}") + return body["data"][0]["id"] + + def chat(self, messages: list[dict], *, max_tokens: int = 256, tools: list | None = None, + temperature: float = 0.0, stream: bool = True, extra: dict | None = None) -> dict[str, Any]: + """Send one chat request and return timings plus the answer. + + Times are seconds from sending the request: ``ttft`` (first answer + text), ``first_reasoning`` (first thinking text, if any), + ``first_sentence`` (first complete sentence of the answer), ``total``. + """ + body: dict[str, Any] = {"model": self.model, "messages": messages, "max_tokens": max_tokens, + "temperature": temperature, "stream": stream} + if stream: + body["stream_options"] = {"include_usage": True} + if tools: + body["tools"] = tools + body.update(self.extra) + if extra: + body.update(extra) + req = urllib.request.Request(f"{self.base}/chat/completions", data=json.dumps(body).encode("utf-8"), + method="POST") + req.add_header("Content-Type", "application/json") + if self.key: + req.add_header("Authorization", f"Bearer {self.key}") + out: dict[str, Any] = {"ttft": None, "first_reasoning": None, "first_sentence": None, "first_tool": None} + content, reasoning = [], [] + tool_calls: dict[int, dict[str, Any]] = {} + usage, finish, last_chunk = None, None, None + t0 = time.perf_counter() + try: + resp = urllib.request.urlopen(req, timeout=self.timeout) + except urllib.error.HTTPError as e: + raise RuntimeError(f"HTTP {e.code}: {e.read().decode('utf-8', 'replace')[:500]}") from None + with resp: + if not stream: + body_out = json.loads(resp.read().decode("utf-8")) + out["total"] = time.perf_counter() - t0 + msg = body_out["choices"][0]["message"] + content = [msg.get("content") or ""] + reasoning = [msg.get("reasoning_content") or ""] + for i, tc in enumerate(msg.get("tool_calls") or []): + tool_calls[i] = {"name": tc["function"]["name"], "arguments": tc["function"].get("arguments", "")} + usage, finish, last_chunk = body_out.get("usage"), body_out["choices"][0].get("finish_reason"), body_out + else: + for raw in resp: + line = raw.decode("utf-8", "replace").strip() + if not line.startswith("data:"): + continue + payload = line[5:].strip() + if payload == "[DONE]": + break + try: + chunk = json.loads(payload) + except ValueError: + continue + last_chunk = chunk + if chunk.get("usage"): + usage = chunk["usage"] + for ch in chunk.get("choices") or []: + d = ch.get("delta") or {} + now = time.perf_counter() - t0 + if d.get("reasoning_content"): + if out["first_reasoning"] is None: + out["first_reasoning"] = now + reasoning.append(d["reasoning_content"]) + if d.get("content"): + if out["ttft"] is None: + out["ttft"] = now + content.append(d["content"]) + if out["first_sentence"] is None and _SENTENCE_END.search("".join(content)): + out["first_sentence"] = now + for tc in d.get("tool_calls") or []: + if out["first_tool"] is None: + out["first_tool"] = now + slot = tool_calls.setdefault(tc.get("index", 0), {"name": "", "arguments": ""}) + fn = tc.get("function") or {} + if fn.get("name"): + slot["name"] = fn["name"] + if fn.get("arguments"): + slot["arguments"] += fn["arguments"] + if ch.get("finish_reason"): + finish = ch["finish_reason"] + out["total"] = time.perf_counter() - t0 + text = "".join(content) + if out["first_sentence"] is None and text: + out["first_sentence"] = out["total"] + out.update({"text": text, "reasoning_chars": len("".join(reasoning)), "finish": finish, "usage": usage, + "tool_calls": [tool_calls[k] for k in sorted(tool_calls)]}) + comp = (usage or {}).get("completion_tokens") + first = out["ttft"] if out["ttft"] is not None else out["first_reasoning"] + if comp and first is not None and out["total"] > first and comp > 1: + out["decode_tok_s"] = round((comp - 1) / (out["total"] - first), 1) + if last_chunk and isinstance(last_chunk, dict): + for k in ("timings", "strata"): + if k in last_chunk: + out[f"server_{k}"] = last_chunk[k] + for k in ("ttft", "first_reasoning", "first_sentence", "first_tool", "total"): + if out.get(k) is not None: + out[k] = round(out[k], 3) + return out diff --git a/scripts/m0/m0lib/stt.py b/scripts/m0/m0lib/stt.py new file mode 100644 index 0000000..129e4f2 --- /dev/null +++ b/scripts/m0/m0lib/stt.py @@ -0,0 +1,170 @@ +"""Stage: faster-whisper large-v3-turbo speed on the GPU. + +Runs inside its own venv (faster-whisper, nvidia-cublas-cu12, nvidia-cudnn-cu12). +Audio: the synthetic Korean samples in assets/ko_samples (MeloTTS voice). +""" + +from __future__ import annotations + +import glob +import json +import os +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from m0lib.common import GpuSampler, StageResult, cer, gpu_memory_mib, say, stage_args, stats # noqa: E402 + +SR = 16000 +ASSETS = Path(__file__).resolve().parent.parent / "assets" / "ko_samples" + + +def add_cuda_dll_dirs() -> list[str]: + """ctranslate2 loads cuBLAS/cuDNN from the PATH on Windows; the pip wheels put them in nvidia/*/bin.""" + found = [] + for sp in sys.path: + for d in glob.glob(os.path.join(sp, "nvidia", "*", "bin")): + if os.path.isdir(d) and d not in found: + found.append(d) + for d in found: + os.environ["PATH"] = d + os.pathsep + os.environ.get("PATH", "") + if hasattr(os, "add_dll_directory"): + os.add_dll_directory(d) + return found + + +def read_wav(path: Path): + """The kit's samples are 16 kHz mono PCM16. Read them with the standard library: + faster-whisper's PyAV path broke on Windows with a newer PyAV + (open() got an unexpected keyword argument 'metadata_errors').""" + import wave + import numpy as np + with wave.open(str(path), "rb") as w: + assert w.getframerate() == SR and w.getnchannels() == 1 and w.getsampwidth() == 2, path + return np.frombuffer(w.readframes(w.getnframes()), dtype=" tuple[str, float]: + t0 = time.perf_counter() + segs, _ = model.transcribe(audio, language="ko", beam_size=beam, vad_filter=False, + condition_on_previous_text=False, without_timestamps=True, hotwords=hotwords) + text = "".join(s.text for s in segs).strip() + return text, time.perf_counter() - t0 + + +def batch_transcribe(model, audios: list, beam: int = 1) -> tuple[list[str], float]: + """One encoder + decoder pass for several speakers' clips (the v2 design: one shared model).""" + import numpy as np + from faster_whisper.tokenizer import Tokenizer + t0 = time.perf_counter() + feats = [] + for a in audios: + a30 = np.pad(a[: SR * 30], (0, max(0, SR * 30 - len(a)))) + feats.append(model.feature_extractor(a30, padding=0)[:, :3000]) + enc = model.encode(np.stack(feats).astype(np.float32)) + tok = Tokenizer(model.hf_tokenizer, model.model.is_multilingual, task="transcribe", language="ko") + prompt = model.get_prompt(tok, previous_tokens=[], without_timestamps=True) + res = model.model.generate(enc, [prompt] * len(audios), beam_size=beam, max_length=224, suppress_blank=True, + suppress_tokens=[-1]) + texts = [tok.decode(x.sequences_ids[0]).strip() for x in res] + return texts, time.perf_counter() - t0 + + +def main() -> int: + out, work, opts = stage_args() + r = StageResult("stt", out) + r["cuda_dll_dirs"] = len(add_cuda_dll_dirs()) + os.environ.setdefault("HF_HOME", str(work / "models" / "hf")) + try: + from faster_whisper import WhisperModel + import ctranslate2 + r["versions"] = {"ctranslate2": ctranslate2.__version__, + "faster_whisper": __import__("faster_whisper").__version__, + "cuda_devices": ctranslate2.get_cuda_device_count()} + except Exception as e: + r.error("import", e) + r.save("failed") + return 0 + + refs = json.loads((ASSETS / "refs.json").read_text(encoding="utf-8"))["samples"] + audio = {s["file"]: read_wav(ASSETS / s["file"]) for s in refs} + name = opts.get("model", "large-v3-turbo") + r["model"] = name + r["audio_source"] = "synthetic MeloTTS Korean (clean speech; real voices will be harder)" + + for ctype in opts.get("compute_types", "int8_float16,float16").split(","): + res: dict = {} + r[f"run_{ctype}"] = res + say(f" - {name} / {ctype}") + try: + before = gpu_memory_mib() + t0 = time.perf_counter() + model = WhisperModel(name, device="cuda", compute_type=ctype, download_root=str(work / "models" / "stt")) + res["load_s"] = round(time.perf_counter() - t0, 2) + transcribe(model, audio[refs[0]["file"]], 1) # warm-up (CUDA kernels) + after = gpu_memory_mib() + if before and after: + res["vram_model_mib"] = after[0] - before[0] + say(f" 로딩 {res['load_s']}s, VRAM 약 {res.get('vram_model_mib')} MiB") + except Exception as e: + r.error(f"load {ctype}", e) + continue + + # Final transcription, fast (beam 1) and accurate (beam 3). + for beam in (1, 3): + rows = [] + for s in refs: + text, sec = transcribe(model, audio[s["file"]], beam) + rows.append({"file": s["file"], "audio_s": s["seconds"], "seconds": round(sec, 3), + "cer": cer(s["text"], text), "text": text}) + res[f"final_beam{beam}"] = {"latency": stats(x["seconds"] for x in rows), + "cer": stats([x["cer"] for x in rows], 4), "rows": rows} + say(f" 최종 전사 beam {beam}: 중앙값 {res[f'final_beam{beam}']['latency']['median']}s, " + f"CER {res[f'final_beam{beam}']['cer']['median']}") + + # Wake-word hint. + rows = [] + for s in refs: + if "자비스" in s["text"]: + plain, _ = transcribe(model, audio[s["file"]], 1) + hinted, _ = transcribe(model, audio[s["file"]], 1, hotwords="자비스") + rows.append({"file": s["file"], "plain": plain, "hotword": hinted, + "plain_has": "자비스" in plain, "hotword_has": "자비스" in hinted}) + res["wake_word"] = rows + + # Streaming: re-transcribe the growing buffer every 0.5 s (beam 1). + long = max(refs, key=lambda s: s["seconds"]) + a = audio[long["file"]] + lat = [] + for end in range(SR // 2, len(a) + 1, SR // 2): + _, sec = transcribe(model, a[:end], 1) + lat.append(round(sec, 3)) + res["partial_every_0_5s"] = {"audio_s": long["seconds"], "latency": stats(lat), "all": lat} + # The last piece only (what v2 decodes after the end of speech). + _, sec = transcribe(model, a[-SR * 2:], 1) + res["last_2s_piece_s"] = round(sec, 3) + + # One shared model for several speakers at once. + batch = {} + with GpuSampler(interval=0.2) as g: + for n in (1, 4, 10): + clips = [audio[refs[i % len(refs)]["file"]] for i in range(n)] + try: + batch_transcribe(model, clips) # warm-up for this batch size + texts, sec = batch_transcribe(model, clips) + errs = [cer(refs[i % len(refs)]["text"], t) for i, t in enumerate(texts)] + batch[str(n)] = {"seconds": round(sec, 3), "cer": stats(errs, 4), "sample_text": texts[0]} + say(f" {n}명 동시(배치): {batch[str(n)]['seconds']}s") + except Exception as e: + r.error(f"batch {n}", e) + batch.update(g.summary()) + res["batch_speakers"] = batch + del model + r.save() + r.save("ok" if not r["errors"] else "partial") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/m0/m0lib/sysinfo.py b/scripts/m0/m0lib/sysinfo.py new file mode 100644 index 0000000..1f9636a --- /dev/null +++ b/scripts/m0/m0lib/sysinfo.py @@ -0,0 +1,157 @@ +"""Stage: PC information (GPU, driver, CPU, RAM, disk, monitors).""" + +from __future__ import annotations + +import ctypes +import json +import os +import platform +import shutil +import subprocess +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from m0lib.common import IS_WINDOWS, StageResult, nvidia_smi, say, stage_args # noqa: E402 + + +def _ps_json(script: str) -> object: + out = subprocess.run(["powershell", "-NoProfile", "-NonInteractive", "-Command", + f"[Console]::OutputEncoding=[Text.Encoding]::UTF8; {script} | ConvertTo-Json -Depth 4"], + capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=60) + text = out.stdout.strip() + return json.loads(text) if text else None + + +def _nvidia() -> dict: + exe = nvidia_smi() + if not exe: + return {"found": False} + fields = ["name", "driver_version", "memory.total", "memory.used", "pcie.link.gen.max", "pcie.link.width.max", + "power.limit", "compute_cap"] + info: dict = {"found": True} + try: + line = subprocess.run([exe, f"--query-gpu={','.join(fields)}", "--format=csv,noheader,nounits"], + capture_output=True, text=True, timeout=20).stdout.strip().splitlines()[0] + info.update({k: v.strip() for k, v in zip(fields, line.split(","))}) + except Exception as e: + info["query_error"] = str(e) + try: + head = subprocess.run([exe], capture_output=True, text=True, timeout=20).stdout + for part in head.split("|"): + if "CUDA Version" in part: + info["cuda_version"] = part.split("CUDA Version:")[1].strip().split()[0] + except Exception: + pass + try: + major = int(str(info.get("driver_version", "0")).split(".")[0]) + info["driver_ok_for_cuda13"] = major >= 580 + except ValueError: + pass + return info + + +def _monitors() -> list[dict]: + if not IS_WINDOWS: + return [] + from ctypes import wintypes + user32 = ctypes.windll.user32 + try: + ctypes.windll.shcore.SetProcessDpiAwareness(2) + except Exception: + pass + shcore = ctypes.windll.shcore + + class MONITORINFOEXW(ctypes.Structure): + _fields_ = [("cbSize", wintypes.DWORD), ("rcMonitor", wintypes.RECT), ("rcWork", wintypes.RECT), + ("dwFlags", wintypes.DWORD), ("szDevice", wintypes.WCHAR * 32)] + + # Declare handle types: a 64-bit HMONITOR passed as a plain int overflows. + user32.GetMonitorInfoW.argtypes = [wintypes.HMONITOR, ctypes.c_void_p] + shcore.GetDpiForMonitor.argtypes = [wintypes.HMONITOR, ctypes.c_int, ctypes.POINTER(ctypes.c_uint), + ctypes.POINTER(ctypes.c_uint)] + mons: list[dict] = [] + proc_t = ctypes.WINFUNCTYPE(ctypes.c_int, wintypes.HMONITOR, wintypes.HDC, ctypes.POINTER(wintypes.RECT), + wintypes.LPARAM) + + def cb(hmon, _hdc, _rect, _lp): + mi = MONITORINFOEXW() + mi.cbSize = ctypes.sizeof(MONITORINFOEXW) + user32.GetMonitorInfoW(hmon, ctypes.byref(mi)) + dx, dy = ctypes.c_uint(), ctypes.c_uint() + try: + shcore.GetDpiForMonitor(hmon, 0, ctypes.byref(dx), ctypes.byref(dy)) + except Exception: + dx.value = 96 + r = mi.rcMonitor + mons.append({"index": len(mons) + 1, "x": r.left, "y": r.top, "width": r.right - r.left, + "height": r.bottom - r.top, "primary": bool(mi.dwFlags & 1), + "scale_percent": round(dx.value / 96 * 100)}) + return 1 + + user32.EnumDisplayMonitors(None, None, proc_t(cb), 0) + return mons + + +def _discord() -> dict: + base = Path(os.environ.get("LOCALAPPDATA", "")) / "Discord" + if not base.exists(): + return {"installed": False} + versions = sorted(p.name for p in base.glob("app-*")) + return {"installed": True, "version": versions[-1][4:] if versions else None} + + +def _hags() -> object: + if not IS_WINDOWS: + return None + import winreg + try: + with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r"SYSTEM\CurrentControlSet\Control\GraphicsDrivers") as k: + v, _ = winreg.QueryValueEx(k, "HwSchMode") + return {1: "off", 2: "on"}.get(v, v) + except OSError: + return "unknown" + + +def main() -> int: + out, work, _ = stage_args() + r = StageResult("sysinfo", out) + r["os"] = {"platform": platform.platform(), "release": platform.release(), "version": platform.version(), + "python": sys.version.split()[0]} + r["gpu"] = _nvidia() + say(f" GPU: {r['gpu'].get('name')} / 드라이버 {r['gpu'].get('driver_version')}") + du = shutil.disk_usage(work) + r["workdir_disk"] = {"total_gb": round(du.total / 1e9), "free_gb": round(du.free / 1e9)} + say(f" 작업 폴더 디스크 남은 공간: {r['workdir_disk']['free_gb']} GB") + if IS_WINDOWS: + for key, script in { + "cpu": "Get-CimInstance Win32_Processor | Select-Object Name,NumberOfCores,NumberOfLogicalProcessors,MaxClockSpeed", + "ram_modules": "Get-CimInstance Win32_PhysicalMemory | Select-Object Capacity,Speed,ConfiguredClockSpeed", + "os_detail": "Get-CimInstance Win32_OperatingSystem | Select-Object Caption,Version,BuildNumber,TotalVisibleMemorySize", + "disks": "Get-PhysicalDisk | Select-Object FriendlyName,MediaType,BusType,Size", + "video_controllers": "Get-CimInstance Win32_VideoController | Select-Object Name,DriverVersion,CurrentHorizontalResolution,CurrentVerticalResolution,CurrentRefreshRate", + }.items(): + try: + r[key] = _ps_json(script) + except Exception as e: + r.error(key, e) + try: + mods = r.get("ram_modules") or [] + mods = mods if isinstance(mods, list) else [mods] + r["ram_total_gb"] = round(sum(int(m.get("Capacity") or 0) for m in mods) / 2**30) + say(f" RAM: {r['ram_total_gb']} GB") + except Exception as e: + r.error("ram_total", e) + try: + r["monitors"] = _monitors() + say(f" 모니터: {len(r['monitors'])}대") + except Exception as e: + r.error("monitors", e) + r["hardware_gpu_scheduling"] = _hags() + r["discord"] = _discord() + r.save("ok" if not r["errors"] else "partial") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/m0/m0lib/tts_cosy.py b/scripts/m0/m0lib/tts_cosy.py new file mode 100644 index 0000000..188670b --- /dev/null +++ b/scripts/m0/m0lib/tts_cosy.py @@ -0,0 +1,135 @@ +"""Stage: CosyVoice2 (and Fun-CosyVoice3) Korean speed and samples. + +Runs in its own Python 3.10 venv. The CosyVoice source (pinned commit) and its +Matcha-TTS submodule are unpacked by the orchestrator into /src. +The voice prompt is one of the kit's synthetic Korean samples, so the output +voice resembles MeloTTS; this measures speed and Korean pronunciation, not timbre. +""" + +from __future__ import annotations + +import os +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from m0lib.common import StageResult, gpu_memory_mib, say, stage_args, stats # noqa: E402 + +ASSETS = Path(__file__).resolve().parent.parent / "assets" / "ko_samples" +MODELS = { + "cosyvoice2": ("FunAudioLLM/CosyVoice2-0.5B", ""), + "cosyvoice3": ("FunAudioLLM/Fun-CosyVoice3-0.5B-2512", "You are a helpful assistant.<|endofprompt|>"), +} +TEXTS = ["네, 알겠습니다. 지금 바로 틀어 드릴게요.", + "오늘 서울은 맑고 낮 최고 기온은 십삼 도까지 오르겠습니다.", + "회의는 오후 세 시에 시작해서 다섯 시쯤 끝날 예정이고, 끝난 뒤에는 저녁 메뉴를 같이 정하기로 하셨습니다."] + + +def patch_load_wav() -> None: + """torchaudio 2.9+ needs torchcodec for load(); read prompts with soundfile instead.""" + import numpy as np + import soundfile as sf + import torch + import torchaudio + import cosyvoice.utils.file_utils as fu + import cosyvoice.cli.frontend as fe + + def load_wav(wav, target_sr, min_sr=16000): + data, sr = sf.read(wav, dtype="float32", always_2d=True) + speech = torch.from_numpy(np.ascontiguousarray(data.mean(axis=1)))[None, :] + if sr != target_sr: + assert sr >= min_sr, f"wav sample rate {sr} must be at least {min_sr}" + speech = torchaudio.transforms.Resample(orig_freq=sr, new_freq=target_sr)(speech) + return speech + + fu.load_wav = load_wav + fe.load_wav = load_wav + + +def main() -> int: + out, work, opts = stage_args() + r = StageResult("tts_cosy", out) + src = Path(opts.get("src", str(work / "src" / "CosyVoice"))) + sys.path[:0] = [str(src), str(src / "third_party" / "Matcha-TTS")] + os.environ.setdefault("HF_HOME", str(work / "models" / "hf")) + os.environ.setdefault("MODELSCOPE_CACHE", str(work / "models" / "modelscope")) + audio_dir = out.parent / "audio" + audio_dir.mkdir(parents=True, exist_ok=True) + try: + import numpy as np + import soundfile as sf + import torch + from huggingface_hub import snapshot_download + from cosyvoice.cli.cosyvoice import AutoModel + patch_load_wav() + r["versions"] = {"torch": torch.__version__, "cuda_available": torch.cuda.is_available()} + except Exception as e: + r.error("import", e) + r.save("failed") + return 0 + + prompt_wav = str(ASSETS / "ko_08.wav") + prompt_text = "오늘 회의는 오후 세 시에 시작해서 다섯 시쯤 끝날 것 같아요. 끝나고 나서 저녁 메뉴를 같이 정해 봐요." + for key in opts.get("models", "cosyvoice2,cosyvoice3").split(","): + repo, prefix = MODELS[key] + res: dict = {"repo": repo} + r[key] = res + say(f" - {key}: 모델 받는 중") + try: + t0 = time.perf_counter() + local = snapshot_download(repo, local_dir=str(work / "models" / "tts" / key)) + res["download_s"] = round(time.perf_counter() - t0, 1) + before = gpu_memory_mib() + t0 = time.perf_counter() + model = AutoModel(model_dir=local, fp16=torch.cuda.is_available()) + res["load_s"] = round(time.perf_counter() - t0, 2) + after = gpu_memory_mib() + if before and after: + res["vram_model_mib"] = after[0] - before[0] + sr = model.sample_rate + # Warm-up so CUDA start-up is not counted. + for _ in model.inference_zero_shot("워밍업입니다.", prefix + prompt_text, prompt_wav, stream=False): + pass + say(f" 로딩 {res['load_s']}s, VRAM 약 {res.get('vram_model_mib')} MiB") + except Exception as e: + r.error(f"{key} load", e) + r.save() + continue + rows = [] + for ti, text in enumerate(TEXTS): + for i in range(2): + try: + if torch.cuda.is_available(): + torch.cuda.synchronize() + t0 = time.perf_counter() + first, chunks = None, [] + for o in model.inference_zero_shot(text, prefix + prompt_text, prompt_wav, stream=True): + if first is None: + first = time.perf_counter() - t0 + chunks.append(o["tts_speech"].squeeze(0).float().cpu().numpy()) + total = time.perf_counter() - t0 + wav = np.concatenate(chunks) if chunks else np.zeros(1, dtype=np.float32) + dur = len(wav) / sr + rows.append({"text": ti, "chars": len(text), "first_chunk_s": round(first or total, 3), + "total_s": round(total, 3), "audio_s": round(dur, 2), + "rtf": round(total / max(dur, 1e-3), 3), "try": i + 1}) + if i == 0 and ti > 0: + sf.write(str(audio_dir / f"{key}_{ti}.wav"), wav, sr) + except Exception as e: + r.error(f"{key} synth {ti}", e) + res["rows"] = rows + res["first_chunk"] = stats(x["first_chunk_s"] for x in rows) + res["rtf"] = stats(x["rtf"] for x in rows) + say(f" 첫 소리까지 중앙값 {res['first_chunk'].get('median')}s, RTF {res['rtf'].get('median')}") + del model + if torch.cuda.is_available(): + torch.cuda.empty_cache() + r.save() + r["audio_files"] = sorted(p.name for p in audio_dir.glob("cosyvoice*.wav")) + r.save("ok" if not r["errors"] else "partial") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/m0/m0lib/tts_melo.py b/scripts/m0/m0lib/tts_melo.py new file mode 100644 index 0000000..fa03b16 --- /dev/null +++ b/scripts/m0/m0lib/tts_melo.py @@ -0,0 +1,148 @@ +"""Stage: MeloTTS Korean speed on the GPU (runs in its own Python 3.11 venv).""" + +from __future__ import annotations + +import os +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from m0lib.common import StageResult, gpu_memory_mib, say, stage_args, stats # noqa: E402 + +SENTENCES = { + "short": ["네, 알겠습니다.", "지금 바로 틀어 드릴게요.", "서울은 지금 맑습니다."], + "medium": ["오늘 서울은 맑고 낮 최고 기온은 십삼 도까지 오르겠습니다.", + "내일 아침 일곱 시에 알람을 맞춰 두었습니다. 푹 주무세요.", + "두 번째 모니터에는 메모장과 크롬 창이 함께 떠 있습니다."], + "long": ["회의는 오후 세 시에 시작해서 다섯 시쯤 끝날 예정이고, 끝난 뒤에는 저녁 메뉴를 같이 정하기로 하셨습니다. " + "필요하시면 회의 삼십 분 전에 한 번 더 알려 드릴게요."], +} + + +def prepare_nltk(folder: Path) -> list[str]: + """Fetch the English g2p data MeloTTS imports into the work folder (not the user profile).""" + import nltk + folder.mkdir(parents=True, exist_ok=True) + os.environ["NLTK_DATA"] = str(folder) + nltk.data.path.insert(0, str(folder)) + got = [] + for pkg, path in (("cmudict", "corpora/cmudict"), ("averaged_perceptron_tagger_eng", "taggers/averaged_perceptron_tagger_eng"), + ("averaged_perceptron_tagger", "taggers/averaged_perceptron_tagger")): + if not (folder / (path + ".zip")).exists() and not (folder / path).exists(): + nltk.download(pkg, download_dir=str(folder), quiet=True) + got.append(pkg) + if os.name == "posix": # newer nltk refuses data folders that others can write to + for root, dirs, files in os.walk(folder): + for n in dirs + files: + p = os.path.join(root, n) + os.chmod(p, os.stat(p).st_mode & ~0o022) + os.chmod(folder, os.stat(folder).st_mode & ~0o022) + return got + + +def korean_only_patches() -> list[str]: + """Make MeloTTS Korean work without the Japanese MeCab, on Windows too. + + melo imports its Japanese front end (module-level ``MeCab.Tagger()``) even + for Korean, and on Windows g2pkk asks for the ``eunjeon`` package instead of + python-mecab-ko. Both are replaced here; v2 will need the same. + """ + import importlib + import types + done = [] + try: + import MeCab # noqa: F401 + MeCab.Tagger # noqa: B018 + except Exception: + stub = types.ModuleType("MeCab") + + class Tagger: # Japanese text is never synthesised in M0 + def __init__(self, *a, **k): + pass + + def parse(self, text): + raise RuntimeError("Japanese MeCab is not installed") + + stub.Tagger = Tagger + sys.modules["MeCab"] = stub + done.append("MeCab stub") + import g2pkk.g2pkk as g2p_mod + if g2p_mod.platform.system() == "Windows": + g2p_mod.G2p.check_mecab = lambda self: None + g2p_mod.G2p.get_mecab = lambda self: importlib.import_module("mecab").MeCab() + done.append("g2pkk uses python-mecab-ko") + return done + + +def main() -> int: + out, work, opts = stage_args() + r = StageResult("tts_melo", out) + os.environ.setdefault("HF_HOME", str(work / "models" / "hf")) + audio_dir = out.parent / "audio" + audio_dir.mkdir(parents=True, exist_ok=True) + try: + r["nltk"] = prepare_nltk(work / "models" / "nltk_data") + r["patches"] = korean_only_patches() + import numpy as np + import soundfile as sf + import torch + from melo.api import TTS + r["versions"] = {"torch": torch.__version__, "cuda": torch.version.cuda, + "cuda_available": torch.cuda.is_available(), + "gpu": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None} + except Exception as e: + r.error("import", e) + r.save("failed") + return 0 + device = "cuda" if torch.cuda.is_available() else "cpu" + try: + before = gpu_memory_mib() + t0 = time.perf_counter() + model = TTS(language="KR", device=device) + spk = model.hps.data.spk2id["KR"] + r["load_s"] = round(time.perf_counter() - t0, 2) + t0 = time.perf_counter() + model.tts_to_file("워밍업입니다.", spk, None, speed=1.0, quiet=True) + r["first_synth_s"] = round(time.perf_counter() - t0, 2) + after = gpu_memory_mib() + if before and after: + r["vram_model_mib"] = after[0] - before[0] + sr = model.hps.data.sampling_rate + say(f" 로딩 {r['load_s']}s, 첫 합성(워밍업) {r['first_synth_s']}s, VRAM 약 {r.get('vram_model_mib')} MiB") + except Exception as e: + r.error("load", e) + r.save("failed") + return 0 + + for speed in (1.0, 1.5): + rows = [] + for kind, texts in SENTENCES.items(): + for text in texts: + for i in range(2): + if device == "cuda": + torch.cuda.synchronize() + t0 = time.perf_counter() + wav = model.tts_to_file(text, spk, None, speed=speed, quiet=True) + if device == "cuda": + torch.cuda.synchronize() + sec = time.perf_counter() - t0 + dur = len(wav) / sr + rows.append({"kind": kind, "chars": len(text), "seconds": round(sec, 3), + "audio_s": round(dur, 2), "rtf": round(sec / dur, 3), "try": i + 1}) + if kind != "short": + name = f"melo_{kind}_{str(speed).replace('.', '_')}.wav" + sf.write(str(audio_dir / name), np.asarray(wav, dtype=np.float32), sr) + r[f"speed_{speed}"] = {"rows": rows, **{k: stats(x["seconds"] for x in rows if x["kind"] == k) + for k in SENTENCES}, + "rtf": stats(x["rtf"] for x in rows)} + say(f" 속도 {speed}: 짧은 문장 {r[f'speed_{speed}']['short'].get('median')}s, " + f"보통 문장 {r[f'speed_{speed}']['medium'].get('median')}s") + r["device"] = device + r["audio_files"] = sorted(p.name for p in audio_dir.glob("melo_*.wav")) + r.save("ok" if not r["errors"] else "partial") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/m0/m0lib/winchecks.py b/scripts/m0/m0lib/winchecks.py new file mode 100644 index 0000000..f75ab66 --- /dev/null +++ b/scripts/m0/m0lib/winchecks.py @@ -0,0 +1,403 @@ +"""Stage: Windows checks that need the user at the PC. + + botwin bot browser window: hidden from Alt+Tab/taskbar but still listed in + Discord's screen-share picker, and still drawing while covered + discord a popped-out Discord stream window: how to recognise it, and + whether Discord stops drawing it while it is covered + games installed games and anti-cheat, and the borderless-window detector + on a game the user starts + +Privacy: window titles of other programs are not stored. The Discord pop-out +title is stored only if the user agrees (it can contain a friend's name). +Game names and exe names are stored; folders are reduced to the launcher name. +""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from m0lib.common import IS_WINDOWS, StageResult, ask, say, stage_args, step, wait_enter # noqa: E402 + +TEST_TITLE = "Jarvis 브라우저 테스트" + +ANTICHEAT_PROCS = { + "vgc.exe": "Vanguard", "vgtray.exe": "Vanguard", + "easyanticheat.exe": "Easy Anti-Cheat", "easyanticheat_eos.exe": "Easy Anti-Cheat (EOS)", + "beservice.exe": "BattlEye", "beservice_x64.exe": "BattlEye", + "faceit.exe": "FACEIT", "faceitservice.exe": "FACEIT", + "gamemon.des": "nProtect GameGuard", "gamemon64.des": "nProtect GameGuard", "npggnt.des": "nProtect GameGuard", + "xxd-0.xem": "XIGNCODE3", "x3.xem": "XIGNCODE3", + "sguard64.exe": "ACE (Tencent)", "sguardsvc64.exe": "ACE (Tencent)", "ace-tray.exe": "ACE (Tencent)", + "eaanticheat.gameservice.exe": "EA AntiCheat", "eaanticheat.gameservicelauncher.exe": "EA AntiCheat", + "pnkbstra.exe": "PunkBuster", "pnkbstrb.exe": "PunkBuster", + "ricochet": "Ricochet", +} +ANTICHEAT_SERVICES = ["vgc", "vgk", "EasyAntiCheat", "EasyAntiCheat_EOS", "BEService", "BEDaisy", "FACEIT", + "FACEITService", "npggsvc", "xhunter1", "ACE-BASE", "AntiCheatExpert Service", "ACE-GAME", + "EAAntiCheatService", "PnkBstrA", "mhyprot3", "zksvc", "atvi-randgrid_sr"] +RIOT_GAME_EXES = {"valorant-win64-shipping.exe": "VALORANT", "league of legends.exe": "League of Legends"} +ANTICHEAT_MARKERS = {"easyanticheat": "Easy Anti-Cheat", "battleye": "BattlEye", "gameguard": "nProtect GameGuard", + "xigncode": "XIGNCODE3", "anticheatexpert": "ACE (Tencent)", "eaanticheat": "EA AntiCheat"} + + +def html_page(path: Path) -> Path: + path.write_text(f"""{TEST_TITLE} + +
""", encoding="utf-8") + return path + + +def find_browser() -> str | None: + cands = [os.path.expandvars(p) for p in ( + r"%ProgramFiles(x86)%\Microsoft\Edge\Application\msedge.exe", + r"%ProgramFiles%\Microsoft\Edge\Application\msedge.exe", + r"%ProgramFiles%\Google\Chrome\Application\chrome.exe", + r"%LOCALAPPDATA%\Google\Chrome\Application\chrome.exe")] + return next((p for p in cands if os.path.exists(p)), None) + + +class Cover: + """An opaque always-on-top window placed exactly over another window.""" + + def __init__(self, rect: list[int]): + import tkinter as tk + self.root = tk.Tk() + self.root.overrideredirect(True) + self.root.attributes("-topmost", True) + self.root.configure(bg="#202020") + left, top, right, bottom = rect + pad = 20 + self.root.geometry(f"{right - left + 2 * pad}x{bottom - top + 2 * pad}+{left - pad}+{top - pad}") + tk.Label(self.root, text="M0 가림 테스트 중 (몇 초 뒤 사라집니다)", fg="white", bg="#202020", + font=("Malgun Gothic", 20)).pack(expand=True) + self.pump(0.5) + + def pump(self, seconds: float) -> None: + end = time.time() + seconds + while time.time() < end: + self.root.update() + time.sleep(0.02) + + def close(self) -> None: + self.root.destroy() + + +def frames(hwnd: int, seconds: float, cover: Cover | None = None, interval: float = 0.25) -> dict: + from m0lib import winwin as w + hashes, black = [], 0 + end = time.time() + seconds + while time.time() < end: + h, is_black = w.capture_hash(hwnd) + hashes.append(h) + black += int(is_black) + if cover: + cover.pump(interval) + else: + time.sleep(interval) + distinct = len({h for h in hashes if h}) + return {"captures": len(hashes), "distinct": distinct, "black": black, + "drawing": distinct >= max(2, len(hashes) // 3)} + + +# -------------------------------------------------------------------------- + +def check_botwin(r: StageResult, work: Path) -> None: + from m0lib import winwin as w + step("1/3 봇용 크롬창: Alt+Tab 숨김과 디스코드 화면공유 목록") + res: dict = {} + r["botwin"] = res + browser = find_browser() + if not browser: + r.error("botwin", "Edge/Chrome을 찾지 못했습니다") + return + res["browser"] = os.path.basename(browser) + page = html_page(work / "m0_clock.html") + profile = work / "browser_test_profile" + proc = subprocess.Popen([browser, f"--user-data-dir={profile}", "--no-first-run", "--no-default-browser-check", + "--disable-backgrounding-occluded-windows", "--disable-renderer-backgrounding", + "--disable-background-timer-throttling", "--window-size=900,600", + "--window-position=200,200", f"--app={page.as_uri()}"]) + try: + hwnd = None + for _ in range(60): + time.sleep(0.5) + for win in w.top_windows(): + if TEST_TITLE in win["title"] and win["class"].startswith("Chrome_WidgetWin"): + hwnd = win["hwnd"] + break + if hwnd: + break + if not hwnd: + r.error("botwin", "테스트 창을 찾지 못했습니다") + return + res["visible_normal"] = frames(hwnd, 3) + say(f"\n '{TEST_TITLE}' 창(시계가 움직이는 창)을 띄웠습니다.") + say(" 디스코드에서 '화면 공유' 버튼을 누르고 '창'(애플리케이션) 목록을 열어 보세요. 확인만 하고 공유는 취소하셔도 됩니다.") + res["discord_list_normal"] = ask(f" 일반 창 상태: 목록에 '{TEST_TITLE}'가 보이나요? (s=디스코드 없음/건너뛰기)", + "yns", "s") + + w.set_tool_window(hwnd) + time.sleep(2) + info = w.window_info(hwnd) + res["toolwindow_style_kept"] = bool(info["exstyle"] & w.WS_EX_TOOLWINDOW) + say("\n 같은 창을 '도구 창' 속성으로 바꾸고 맨 뒤로 보냈습니다.") + res["alt_tab_hidden"] = ask(" Alt+Tab을 눌러 보세요. 목록에서 이 창이 빠졌나요?", "yn") + res["taskbar_hidden"] = ask(" 작업 표시줄에서도 이 창 아이콘이 빠졌나요?", "yn") + res["win_tab_hidden"] = ask(" Win+Tab(작업 보기)에서도 빠졌나요?", "yn") + say(" 디스코드 '화면 공유 → 창' 목록을 닫았다가 다시 열어 보세요.") + res["discord_list_toolwindow"] = ask(f" 도구 창 상태: 목록에 '{TEST_TITLE}'가 아직 보이나요? (s=건너뛰기)", + "yns", "s") + if res["discord_list_toolwindow"] == "y": + say(" 그 창을 실제로 화면공유해 보세요(나만 있는 채널도 괜찮습니다). 공유를 시작했으면 Enter.") + if wait_enter(" (건너뛰려면 s 입력 후 Enter)") != "s": + cover = Cover(info["rect"]) + try: + res["covered_capture"] = frames(hwnd, 6, cover) + finally: + cover.close() + res["discord_preview_moving_while_covered"] = ask( + " 방금 창이 가려져 있는 동안 디스코드 화면공유 미리보기의 시계가 계속 움직였나요?", "yns", "s") + if "covered_capture" not in res: + cover = Cover(info["rect"]) + try: + res["covered_capture"] = frames(hwnd, 4, cover) + finally: + cover.close() + say(f" 가려졌을 때 창 그리기 계속: {'예' if res['covered_capture']['drawing'] else '아니요'}") + finally: + w.kill_tree(proc.pid) + w.kill_by_cmdline(profile.name) + + +def check_discord(r: StageResult) -> None: + from m0lib import winwin as w + step("2/3 디스코드 화면공유 창 분리(팝아웃)") + res: dict = {} + r["discord"] = res + procs = w.processes() + pids = {pid for pid, name in procs.items() if name == "discord.exe"} + res["discord_running"] = bool(pids) + if not pids: + say(" 디스코드가 실행 중이 아니라 건너뜁니다.") + return + + def dwins() -> dict[int, dict]: + return {x["hwnd"]: x for x in w.top_windows() if x["pid"] in pids and x["title"]} + + before = dwins() + say(" 친구(또는 다른 기기의 내 계정)의 화면공유를 시청하고, 시청 화면 오른쪽 위의 '창 분리(팝아웃)' 버튼을 눌러 주세요.") + say(" 화면에 계속 움직이는 내용(영상 등)이 나오면 더 정확합니다.") + if wait_enter(" 분리된 창이 떴으면 Enter, 지금 할 수 없으면 s 입력 후 Enter:") == "s": + res["skipped"] = True + return + time.sleep(1) + after = dwins() + new = [x for h, x in after.items() if h not in before] + res["new_windows"] = len(new) + if not new: + # Already popped out before we looked: take the smaller Discord windows. + new = sorted(after.values(), key=lambda x: (x["rect"][2] - x["rect"][0]) * (x["rect"][3] - x["rect"][1]))[:-1] + res["note"] = "no new window; used existing smaller Discord windows" + if not new: + r.error("discord", "분리된 창을 찾지 못했습니다") + return + pop = new[0] + say(f" 찾은 창 제목: {pop['title']}") + res["title_has_streamer_name"] = ask(" 이 제목에 화면공유한 사람 이름이 들어 있나요?", "yn") + keep = ask(" 이 제목을 결과 파일에 넣어도 될까요? (이름이 들어 있을 수 있습니다)", "yn", "n") + res["title"] = pop["title"] if keep == "y" else re.sub(r"\w", "x", pop["title"]) + res["class"] = pop["class"] + res["size"] = [pop["rect"][2] - pop["rect"][0], pop["rect"][3] - pop["rect"][1]] + res["main_window_titles_masked"] = [re.sub(r"\w", "x", x["title"]) for x in after.values() if x is not pop] + say(" 분리된 창이 다른 창에 가려지지 않게 화면에 보이도록 두세요. 5초 동안 확인합니다.") + time.sleep(2) + res["visible_capture"] = frames(pop["hwnd"], 5) + cover = Cover(pop["rect"]) + try: + res["covered_capture"] = frames(pop["hwnd"], 6, cover) + finally: + cover.close() + res["visible_capture_after"] = frames(pop["hwnd"], 3) + say(f" 보일 때 그리기: {'예' if res['visible_capture']['drawing'] else '아니요'}, " + f"가려졌을 때 그리기: {'예' if res['covered_capture']['drawing'] else '아니요'}") + + +def _steam_games() -> list[dict]: + import winreg + out = [] + try: + with winreg.OpenKey(winreg.HKEY_CURRENT_USER, r"Software\Valve\Steam") as k: + steam = Path(winreg.QueryValueEx(k, "SteamPath")[0]) + except OSError: + return out + libs = {steam} + vdf = steam / "steamapps" / "libraryfolders.vdf" + if vdf.exists(): + for m in re.finditer(r'"path"\s+"([^"]+)"', vdf.read_text(encoding="utf-8", errors="replace")): + libs.add(Path(m.group(1).replace("\\\\", "\\"))) + for lib in libs: + for acf in (lib / "steamapps").glob("appmanifest_*.acf"): + t = acf.read_text(encoding="utf-8", errors="replace") + name = re.search(r'"name"\s+"([^"]+)"', t) + inst = re.search(r'"installdir"\s+"([^"]+)"', t) + if name and inst: + out.append({"launcher": "Steam", "name": name.group(1), + "dir": lib / "steamapps" / "common" / inst.group(1)}) + return out + + +def _epic_games() -> list[dict]: + out = [] + man = Path(os.environ.get("ProgramData", r"C:\ProgramData")) / "Epic" / "EpicGamesLauncher" / "Data" / "Manifests" + for item in man.glob("*.item"): + try: + d = json.loads(item.read_text(encoding="utf-8", errors="replace")) + out.append({"launcher": "Epic", "name": d.get("DisplayName"), "dir": Path(d.get("InstallLocation", ""))}) + except Exception: + continue + return out + + +def _riot_games() -> list[dict]: + out = [] + meta = Path(os.environ.get("ProgramData", r"C:\ProgramData")) / "Riot Games" / "Metadata" + for yaml in meta.glob("*/*.product_settings.yaml"): + t = yaml.read_text(encoding="utf-8", errors="replace") + m = re.search(r'product_install_full_path:\s*"?([^"\n]+)', t) + out.append({"launcher": "Riot", "name": yaml.parent.name.split(".")[0], + "dir": Path(m.group(1).strip()) if m else Path("")}) + return out + + +def _anticheat_in_dir(d: Path, budget: int = 4000) -> list[str]: + found: set[str] = set() + if not d or not d.is_dir(): + return [] + seen = 0 + for root, dirs, files in os.walk(d): + depth = len(Path(root).relative_to(d).parts) + if depth >= 3: + dirs[:] = [] + for n in dirs + files: + seen += 1 + low = n.lower() + for marker, label in ANTICHEAT_MARKERS.items(): + if marker in low: + found.add(label) + if low.startswith("beservice") or low == "be": + found.add("BattlEye") + if seen > budget: + break + return sorted(found) + + +def _sample_foreground(w) -> dict: + fg = w.foreground() + if not fg: + return {} + mon = w.monitor_rect(fg["hwnd"]) + path = w.proc_path(fg["pid"]) or "" + exe = os.path.basename(path).lower() + gpu = w.gpu_3d_by_pid() + low_path = path.lower() + launcher = next((n for key, n in (("steamapps", "Steam"), ("epic games", "Epic"), ("riot games", "Riot"), + ("battle.net", "Battle.net"), ("nexon", "Nexon")) if key in low_path), None) + covers = fg["rect"][0] <= mon[0] and fg["rect"][1] <= mon[1] and fg["rect"][2] >= mon[2] and \ + fg["rect"][3] >= mon[3] + no_frame = not (fg["style"] & w.WS_CAPTION) and not (fg["style"] & w.WS_THICKFRAME) + gpu_3d = gpu.get(fg["pid"], 0.0) + return {"exe": exe, "class": fg["class"], "covers_monitor": covers, "no_caption_or_border": no_frame, + "popup_style": bool(fg["style"] & w.WS_POPUP), "gpu_3d_percent": gpu_3d, + "top_gpu_3d": sorted(((v, (w.proc_path(p) and os.path.basename(w.proc_path(p)).lower()) or str(p)) + for p, v in gpu.items() if v > 1), reverse=True)[:5], + "notification_state": w.notification_state(), "launcher_folder": launcher, + "rect": fg["rect"], "monitor": mon, + "detector_verdict": bool(covers and no_frame and gpu_3d >= 10) or w.notification_state() == "d3d_fullscreen" + or launcher is not None} + + +def check_games(r: StageResult) -> None: + from m0lib import winwin as w + step("3/3 게임과 안티치트 감지") + res: dict = {} + r["games"] = res + say(" 설치된 게임과 안티치트를 찾는 중 (게임 파일 이름만 보고, 내용은 읽지 않습니다)") + games = [] + for fn in (_steam_games, _epic_games, _riot_games): + try: + games += fn() + except Exception as e: + r.error(fn.__name__, e) + res["installed"] = [{"launcher": g["launcher"], "name": g["name"], "anticheat_files": _anticheat_in_dir(g["dir"])} + for g in games] + res["anticheat_services"] = w.services_states(ANTICHEAT_SERVICES) + running = w.processes() + res["anticheat_running_now"] = sorted({ANTICHEAT_PROCS[n] for n in running.values() if n in ANTICHEAT_PROCS}) + say(f" 설치된 게임 {len(games)}개, 안티치트 서비스/드라이버 {len(res['anticheat_services'])}개") + + samples = [] + labels = [("game", "게임을 하나 켜 주세요. 테두리 없는 창 모드(borderless)면 가장 좋습니다."), + ("anticheat_game", "안티치트가 있는 게임(예: 발로란트, 배틀그라운드)이 있으면 켜 주세요."), + ("not_game", "게임이 아닌 경우도 봅니다. 브라우저에서 영상을 전체화면으로 틀어 주세요.")] + for kind, guide in labels: + say(f"\n {guide}") + if wait_enter(" 준비되면 Enter (건너뛰려면 s 입력 후 Enter). Enter 뒤 15초 안에 그 화면으로 전환해 주세요:") == "s": + continue + for left in range(15, 0, -5): + say(f" {left}초 뒤 확인합니다...") + time.sleep(5) + rows = [] + for _ in range(3): + rows.append(_sample_foreground(w)) + time.sleep(1) + procs = w.processes() + anti = sorted({ANTICHEAT_PROCS[n] for n in procs.values() if n in ANTICHEAT_PROCS}) + riot = sorted({RIOT_GAME_EXES[n] for n in procs.values() if n in RIOT_GAME_EXES}) + try: + import winsound + winsound.MessageBeep() + except Exception: + pass + samples.append({"kind": kind, "samples": rows, "anticheat_running": anti, "riot_games_running": riot}) + say(f" 확인 끝(삑 소리). 앞 창: {rows[-1].get('exe')}, 게임 판정: {rows[-1].get('detector_verdict')}, " + f"안티치트: {', '.join(anti) or '없음'}") + if kind != "not_game": + samples[-1]["user_says_mode"] = ask(" 그 게임의 화면 모드는? b=테두리 없는 창, f=전체화면, w=창 모드", "bfw", "b") + res["live"] = samples + + +def main() -> int: + out, work, opts = stage_args() + r = StageResult("winchecks", out) + if not IS_WINDOWS: + r.error("winchecks", "Windows가 아니라서 건너뜁니다") + r.save("skipped") + return 0 + from m0lib import winwin as w + w.dpi_aware() + only = set(opts.get("only", "botwin,discord,games").split(",")) + for name, fn in (("botwin", lambda: check_botwin(r, work)), ("discord", lambda: check_discord(r)), + ("games", lambda: check_games(r))): + if name not in only: + continue + try: + fn() + except Exception as e: + r.error(name, e) + r.save() + r.save("ok" if not r["errors"] else "partial") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/m0/m0lib/winwin.py b/scripts/m0/m0lib/winwin.py new file mode 100644 index 0000000..4f6dc77 --- /dev/null +++ b/scripts/m0/m0lib/winwin.py @@ -0,0 +1,268 @@ +"""Small Win32 helpers over ctypes (windows, processes, capture). Windows only.""" + +from __future__ import annotations + +import ctypes +import hashlib +import json +import os +import subprocess +from ctypes import wintypes + +user32 = ctypes.WinDLL("user32", use_last_error=True) +kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) +gdi32 = ctypes.WinDLL("gdi32", use_last_error=True) +psapi = ctypes.WinDLL("psapi", use_last_error=True) +dwmapi = ctypes.WinDLL("dwmapi") +shell32 = ctypes.WinDLL("shell32") + +GWL_STYLE, GWL_EXSTYLE = -16, -20 +WS_CAPTION, WS_THICKFRAME, WS_POPUP = 0x00C00000, 0x00040000, 0x80000000 +WS_EX_TOOLWINDOW, WS_EX_APPWINDOW, WS_EX_TOPMOST = 0x00000080, 0x00040000, 0x00000008 +SW_HIDE, SW_SHOWNOACTIVATE = 0, 4 +SWP_NOSIZE, SWP_NOMOVE, SWP_NOACTIVATE, SWP_FRAMECHANGED = 0x1, 0x2, 0x10, 0x20 +HWND_BOTTOM = wintypes.HWND(1) +PW_RENDERFULLCONTENT = 2 +PROCESS_QUERY_LIMITED_INFORMATION = 0x1000 + +# Every handle argument must be declared: without argtypes ctypes passes a +# Python int as a 32-bit C int, and a 64-bit HWND then fails with +# "OverflowError: int too long to convert" (seen on a real Windows 11 run). +_H, _D, _U, _I = wintypes.HWND, wintypes.DWORD, wintypes.UINT, ctypes.c_int +for _fn, _args, _res in ( + (user32.GetWindowLongPtrW, [_H, _I], ctypes.c_ssize_t), + (user32.SetWindowLongPtrW, [_H, _I, ctypes.c_ssize_t], ctypes.c_ssize_t), + (user32.GetForegroundWindow, [], _H), + (user32.GetWindowThreadProcessId, [_H, ctypes.POINTER(_D)], _D), + (user32.GetWindowTextLengthW, [_H], _I), + (user32.GetWindowTextW, [_H, wintypes.LPWSTR, _I], _I), + (user32.GetClassNameW, [_H, wintypes.LPWSTR, _I], _I), + (user32.GetWindowRect, [_H, ctypes.POINTER(wintypes.RECT)], wintypes.BOOL), + (user32.IsWindowVisible, [_H], wintypes.BOOL), + (user32.IsIconic, [_H], wintypes.BOOL), + (user32.MonitorFromWindow, [_H, _D], wintypes.HMONITOR), + (user32.GetMonitorInfoW, [wintypes.HMONITOR, ctypes.c_void_p], wintypes.BOOL), + (user32.GetWindowDC, [_H], wintypes.HDC), + (user32.ReleaseDC, [_H, wintypes.HDC], _I), + (user32.PrintWindow, [_H, wintypes.HDC, _U], wintypes.BOOL), + (user32.ShowWindow, [_H, _I], wintypes.BOOL), + (user32.SetWindowPos, [_H, _H, _I, _I, _I, _I, _U], wintypes.BOOL), + (gdi32.CreateCompatibleDC, [wintypes.HDC], wintypes.HDC), + (gdi32.CreateCompatibleBitmap, [wintypes.HDC, _I, _I], wintypes.HBITMAP), + (gdi32.SelectObject, [wintypes.HDC, wintypes.HGDIOBJ], wintypes.HGDIOBJ), + (gdi32.GetDIBits, [wintypes.HDC, wintypes.HBITMAP, _U, _U, ctypes.c_void_p, ctypes.c_void_p, _U], _I), + (gdi32.DeleteObject, [wintypes.HGDIOBJ], wintypes.BOOL), + (gdi32.DeleteDC, [wintypes.HDC], wintypes.BOOL), + (kernel32.OpenProcess, [_D, wintypes.BOOL, _D], wintypes.HANDLE), + (kernel32.QueryFullProcessImageNameW, [wintypes.HANDLE, _D, wintypes.LPWSTR, ctypes.POINTER(_D)], wintypes.BOOL), + (kernel32.CloseHandle, [wintypes.HANDLE], wintypes.BOOL), + (psapi.EnumProcesses, [ctypes.POINTER(_D), _D, ctypes.POINTER(_D)], wintypes.BOOL), + (dwmapi.DwmGetWindowAttribute, [_H, _D, ctypes.c_void_p, _D], ctypes.c_long), +): + _fn.argtypes, _fn.restype = _args, _res + +def dpi_aware() -> None: + try: + user32.SetProcessDpiAwarenessContext(ctypes.c_void_p(-4)) # per-monitor v2 + except Exception: + try: + ctypes.windll.shcore.SetProcessDpiAwareness(2) + except Exception: + pass + + +class MONITORINFO(ctypes.Structure): + _fields_ = [("cbSize", wintypes.DWORD), ("rcMonitor", wintypes.RECT), ("rcWork", wintypes.RECT), + ("dwFlags", wintypes.DWORD)] + + +def _rect(r: wintypes.RECT) -> list[int]: + return [r.left, r.top, r.right, r.bottom] + + +def window_info(hwnd: int) -> dict: + pid = wintypes.DWORD() + user32.GetWindowThreadProcessId(hwnd, ctypes.byref(pid)) + n = user32.GetWindowTextLengthW(hwnd) + buf = ctypes.create_unicode_buffer(n + 1) + user32.GetWindowTextW(hwnd, buf, n + 1) + cls = ctypes.create_unicode_buffer(256) + user32.GetClassNameW(hwnd, cls, 256) + r = wintypes.RECT() + user32.GetWindowRect(hwnd, ctypes.byref(r)) + cloaked = ctypes.c_int(0) + dwmapi.DwmGetWindowAttribute(wintypes.HWND(hwnd), 14, ctypes.byref(cloaked), ctypes.sizeof(cloaked)) + return {"hwnd": int(hwnd), "pid": pid.value, "title": buf.value, "class": cls.value, "rect": _rect(r), + "visible": bool(user32.IsWindowVisible(hwnd)), "minimized": bool(user32.IsIconic(hwnd)), + "cloaked": bool(cloaked.value), "style": user32.GetWindowLongPtrW(hwnd, GWL_STYLE) & 0xFFFFFFFF, + "exstyle": user32.GetWindowLongPtrW(hwnd, GWL_EXSTYLE) & 0xFFFFFFFF} + + +def top_windows(visible_only: bool = True) -> list[dict]: + found: list[dict] = [] + proc = ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.HWND, wintypes.LPARAM) + + def cb(hwnd, _lp): + if not visible_only or user32.IsWindowVisible(hwnd): + found.append(window_info(hwnd)) + return True + + user32.EnumWindows(proc(cb), 0) + return found + + +def monitor_rect(hwnd: int) -> list[int]: + hmon = user32.MonitorFromWindow(wintypes.HWND(hwnd), 2) + mi = MONITORINFO() + mi.cbSize = ctypes.sizeof(MONITORINFO) + user32.GetMonitorInfoW(hmon, ctypes.byref(mi)) + return _rect(mi.rcMonitor) + + +def proc_path(pid: int) -> str | None: + h = kernel32.OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, False, pid) + if not h: + return None + try: + buf = ctypes.create_unicode_buffer(1024) + size = wintypes.DWORD(1024) + if kernel32.QueryFullProcessImageNameW(h, 0, buf, ctypes.byref(size)): + return buf.value + return None + finally: + kernel32.CloseHandle(h) + + +def processes() -> dict[int, str]: + """pid -> exe name (lower case).""" + arr = (wintypes.DWORD * 8192)() + needed = wintypes.DWORD() + psapi.EnumProcesses(arr, ctypes.sizeof(arr), ctypes.byref(needed)) + out = {} + for pid in arr[: needed.value // ctypes.sizeof(wintypes.DWORD)]: + if pid: + p = proc_path(pid) + if p: + out[pid] = os.path.basename(p).lower() + return out + + +def foreground() -> dict | None: + h = user32.GetForegroundWindow() + return window_info(h) if h else None + + +def notification_state() -> str: + """SHQueryUserNotificationState: 'd3d_fullscreen' is exclusive fullscreen D3D.""" + v = ctypes.c_int(0) + shell32.SHQueryUserNotificationState(ctypes.byref(v)) + return {1: "not_present", 2: "busy", 3: "d3d_fullscreen", 4: "presentation_mode", 5: "accepts_notifications", + 6: "quiet_time", 7: "app"}.get(v.value, str(v.value)) + + +def capture_hash(hwnd: int) -> tuple[str | None, bool]: + """PrintWindow(PW_RENDERFULLCONTENT) -> (md5 of pixels, all_black). Works for covered windows.""" + r = wintypes.RECT() + user32.GetWindowRect(hwnd, ctypes.byref(r)) + w, h = r.right - r.left, r.bottom - r.top + if w <= 0 or h <= 0: + return None, True + hdc_win = user32.GetWindowDC(hwnd) + hdc = gdi32.CreateCompatibleDC(hdc_win) + bmp = gdi32.CreateCompatibleBitmap(hdc_win, w, h) + old = gdi32.SelectObject(hdc, bmp) + try: + ok = user32.PrintWindow(wintypes.HWND(hwnd), hdc, PW_RENDERFULLCONTENT) + if not ok: + return None, True + + class BITMAPINFOHEADER(ctypes.Structure): + _fields_ = [("biSize", wintypes.DWORD), ("biWidth", wintypes.LONG), ("biHeight", wintypes.LONG), + ("biPlanes", wintypes.WORD), ("biBitCount", wintypes.WORD), + ("biCompression", wintypes.DWORD), ("biSizeImage", wintypes.DWORD), + ("biXPelsPerMeter", wintypes.LONG), ("biYPelsPerMeter", wintypes.LONG), + ("biClrUsed", wintypes.DWORD), ("biClrImportant", wintypes.DWORD)] + + bi = BITMAPINFOHEADER(ctypes.sizeof(BITMAPINFOHEADER), w, -h, 1, 32, 0, 0, 0, 0, 0, 0) + buf = ctypes.create_string_buffer(w * h * 4) + gdi32.GetDIBits(hdc, bmp, 0, h, buf, ctypes.byref(bi), 0) + raw = buf.raw + # Sample every 97th byte for the black check (fast enough for 4K). + black = max(raw[::97]) < 8 + return hashlib.md5(raw).hexdigest(), black + finally: + gdi32.SelectObject(hdc, old) + gdi32.DeleteObject(bmp) + gdi32.DeleteDC(hdc) + user32.ReleaseDC(hwnd, hdc_win) + + +def set_tool_window(hwnd: int) -> None: + """Hide from Alt+Tab / taskbar (tool window) and send to the bottom of the z-order.""" + ex = user32.GetWindowLongPtrW(hwnd, GWL_EXSTYLE) + user32.ShowWindow(hwnd, SW_HIDE) + user32.SetWindowLongPtrW(hwnd, GWL_EXSTYLE, (ex | WS_EX_TOOLWINDOW) & ~WS_EX_APPWINDOW) + user32.ShowWindow(hwnd, SW_SHOWNOACTIVATE) + user32.SetWindowPos(hwnd, HWND_BOTTOM, 0, 0, 0, 0, SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_FRAMECHANGED) + + +def gpu_3d_by_pid() -> dict[int, float]: + """3D engine utilisation per process from the 'GPU Engine' performance counters.""" + script = ("$s=(Get-Counter '\\GPU Engine(*engtype_3D)\\Utilization Percentage' -ErrorAction SilentlyContinue)." + "CounterSamples; $s | Select-Object InstanceName,CookedValue | ConvertTo-Json -Compress") + try: + out = subprocess.run(["powershell", "-NoProfile", "-NonInteractive", "-Command", script], + capture_output=True, text=True, timeout=60).stdout.strip() + rows = json.loads(out) if out else [] + except Exception: + return {} + rows = rows if isinstance(rows, list) else [rows] + usage: dict[int, float] = {} + for row in rows: + name = str(row.get("InstanceName", "")) + if name.startswith("pid_"): + try: + pid = int(name.split("_")[1]) + except ValueError: + continue + usage[pid] = round(usage.get(pid, 0.0) + float(row.get("CookedValue") or 0), 1) + return usage + + +def services_states(names: list[str]) -> dict[str, dict]: + """Installed services and kernel drivers among ``names`` -> {name: {kind, state, start}}. + + Uses CIM so the answer does not depend on the Windows display language. + """ + quoted = ",".join("'" + n.replace("'", "") + "'" for n in names) + script = (f"$n=@({quoted}); " + "$a=@(Get-CimInstance Win32_Service | Where-Object { $n -contains $_.Name } | " + "Select-Object Name,State,StartMode,@{n='Kind';e={'service'}}); " + "$b=@(Get-CimInstance Win32_SystemDriver | Where-Object { $n -contains $_.Name } | " + "Select-Object Name,State,StartMode,@{n='Kind';e={'driver'}}); " + "ConvertTo-Json -Compress -InputObject @($a + $b)") + try: + out = subprocess.run(["powershell", "-NoProfile", "-NonInteractive", "-Command", script], + capture_output=True, text=True, timeout=120).stdout.strip() + rows = json.loads(out) if out else [] + except Exception: + return {} + rows = rows if isinstance(rows, list) else [rows] + return {r["Name"]: {"kind": r.get("Kind"), "state": r.get("State"), "start": r.get("StartMode")} for r in rows} + + +def kill_tree(pid: int) -> None: + subprocess.run(["taskkill", "/PID", str(pid), "/T", "/F"], capture_output=True) + + +def kill_by_cmdline(fragment: str) -> None: + """Stop processes whose command line contains ``fragment``. + + Chromium's first process can hand the window to another process and exit, + so killing by the launched pid alone can leave the window open. + """ + frag = fragment.replace("'", "") + script = (f"Get-CimInstance Win32_Process | Where-Object {{ $_.CommandLine -like '*{frag}*' }} | " + "ForEach-Object { Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue }") + subprocess.run(["powershell", "-NoProfile", "-NonInteractive", "-Command", script], capture_output=True, + timeout=60) diff --git a/scripts/m0/run_m0.bat b/scripts/m0/run_m0.bat new file mode 100644 index 0000000..2d141d5 --- /dev/null +++ b/scripts/m0/run_m0.bat @@ -0,0 +1,5 @@ +@echo off +chcp 65001 >nul +powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0run_m0.ps1" %* +echo. +pause diff --git a/scripts/m0/run_m0.ps1 b/scripts/m0/run_m0.ps1 new file mode 100644 index 0000000..452d5f8 --- /dev/null +++ b/scripts/m0/run_m0.ps1 @@ -0,0 +1,50 @@ +# Jarvis v2 M0 measurement kit - bootstrap (ASCII only on purpose: Windows +# PowerShell 5.1 reads a BOM-less script in the ANSI code page). +# +# Installs uv and Python into the work folder, then runs m0.py. +# Nothing is installed outside the work folder except what Strata's own +# installer needs (it is pointed at the work folder too). +param( + [string]$WorkDir = "", + [string]$Only = "", + [string]$Skip = "", + [switch]$Redo +) +$ErrorActionPreference = 'Stop' +[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 +try { [Console]::OutputEncoding = [Text.UTF8Encoding]::new($false) } catch { } + +$Kit = Split-Path -Parent $MyInvocation.MyCommand.Path +if (-not $WorkDir) { + # A short path on the same drive as the kit: deep venv paths break under MAX_PATH. + $WorkDir = Join-Path ([IO.Path]::GetPathRoot($Kit)) 'JarvisM0' +} +New-Item -ItemType Directory -Force -Path $WorkDir | Out-Null +$WorkDir = (Resolve-Path $WorkDir).Path +Write-Host "[M0] work folder: $WorkDir" + +$env:UV_CACHE_DIR = Join-Path $WorkDir 'cache\uv' +$env:UV_PYTHON_INSTALL_DIR = Join-Path $WorkDir 'python' +$env:UV_PYTHON_PREFERENCE = 'only-managed' +$env:UV_LINK_MODE = 'copy' +$env:PYTHONUTF8 = '1' +$env:PYTHONIOENCODING = 'utf-8' + +$uvDir = Join-Path $WorkDir 'uv' +$uv = Join-Path $uvDir 'uv.exe' +if (-not (Test-Path $uv)) { + Write-Host "[M0] downloading uv" + New-Item -ItemType Directory -Force -Path $uvDir | Out-Null + $zip = Join-Path $WorkDir 'uv.zip' + Invoke-WebRequest -UseBasicParsing -Uri 'https://github.com/astral-sh/uv/releases/download/0.11.20/uv-x86_64-pc-windows-msvc.zip' -OutFile $zip + Expand-Archive -Path $zip -DestinationPath $uvDir -Force + Remove-Item $zip +} +$env:M0_UV = $uv + +$m0args = @('run', '--no-project', '--python', '3.12', (Join-Path $Kit 'm0.py'), '--work', $WorkDir) +if ($Only) { $m0args += @('--only', $Only) } +if ($Skip) { $m0args += @('--skip', $Skip) } +if ($Redo) { $m0args += '--redo' } +& $uv @m0args +exit $LASTEXITCODE diff --git a/tests/test_m0_kit.py b/tests/test_m0_kit.py new file mode 100644 index 0000000..0c18500 --- /dev/null +++ b/tests/test_m0_kit.py @@ -0,0 +1,102 @@ +"""Behaviour of the M0 measurement kit's pure parts (scripts/m0). + +The kit runs once on the gaming PC and its results leave the PC, so the +privacy scrub and the timing client must behave before it is handed over. +""" + +from __future__ import annotations + +import json +import sys +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scripts" / "m0")) + +from m0lib import common # noqa: E402 +from m0lib.llm_client import ChatClient # noqa: E402 + +pytestmark = pytest.mark.unit + + +def test_scrub_masks_home_folder_and_user_name(monkeypatch): + monkeypatch.setattr(common, "_TOKENS", sorted({r"C:\Users\minsu", "minsu", "MINSU-PC"}, key=len, reverse=True)) + text = r"model at C:\Users\minsu\JarvisM0, user minsu on MINSU-PC, json C:\\Users\\minsu\\x" + out = common.scrub(text) + assert "minsu" not in out.lower() + assert "" in out and "" in out + + +def test_scrub_leaves_words_that_contain_a_short_user_name(monkeypatch): + monkeypatch.setattr(common, "_TOKENS", ["ci"]) + assert common.scrub("CUDA driver version is insufficient; user ci") == "CUDA driver version is insufficient; user " + + +def test_written_results_are_scrubbed(tmp_path, monkeypatch): + monkeypatch.setattr(common, "_TOKENS", ["/home/someone", "someone"]) + target = tmp_path / "r.json" + common.write_json(target, {"path": "/home/someone/work", "who": ["someone"]}) + data = json.loads(target.read_text(encoding="utf-8")) + assert data == {"path": "/work", "who": [""]} + + +def test_cer_ignores_spacing_and_punctuation(): + assert common.cer("자비스, 오늘 날씨 어때?", "자비스 오늘 날씨 어때") == 0.0 + assert common.cer("가나다라", "가나다마") == 0.25 + + +class _FakeOpenAI(BaseHTTPRequestHandler): + """Streams a tool call split over chunks, then text, then usage.""" + + def log_message(self, *a): + pass + + def do_GET(self): + body = json.dumps({"data": [{"id": "fake-model"}]}).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_POST(self): + self.rfile.read(int(self.headers.get("Content-Length", 0))) + chunks = [ + {"choices": [{"delta": {"tool_calls": [{"index": 0, "function": {"name": "get_weather", "arguments": ""}}]}}]}, + {"choices": [{"delta": {"tool_calls": [{"index": 0, "function": {"arguments": '{"city": '}}]}}]}, + {"choices": [{"delta": {"tool_calls": [{"index": 0, "function": {"arguments": '"서울"}'}}]}}]}, + {"choices": [{"delta": {"content": "네, 확인해 볼게요. "}}]}, + {"choices": [{"delta": {"content": "잠시만요"}, "finish_reason": "tool_calls"}]}, + {"choices": [], "usage": {"prompt_tokens": 12, "completion_tokens": 9}}, + ] + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.end_headers() + for c in chunks: + self.wfile.write(f"data: {json.dumps(c, ensure_ascii=False)}\n\n".encode()) + self.wfile.flush() + self.wfile.write(b"data: [DONE]\n\n") + + +@pytest.fixture() +def fake_server(): + srv = ThreadingHTTPServer(("127.0.0.1", 0), _FakeOpenAI) + t = threading.Thread(target=srv.serve_forever, daemon=True) + t.start() + yield f"http://127.0.0.1:{srv.server_address[1]}/v1" + srv.shutdown() + + +def test_chat_client_assembles_streamed_tool_call_and_timings(fake_server): + c = ChatClient(fake_server) + assert c.model == "fake-model" + a = c.chat([{"role": "user", "content": "서울 날씨"}], max_tokens=50) + assert a["tool_calls"] == [{"name": "get_weather", "arguments": '{"city": "서울"}'}] + assert a["text"] == "네, 확인해 볼게요. 잠시만요" + assert a["finish"] == "tool_calls" + assert a["usage"]["completion_tokens"] == 9 + # The first sentence ends at "요. ", before the whole answer is done. + assert a["first_tool"] <= a["ttft"] <= a["first_sentence"] <= a["total"]