feat(m0): measure a YouTube video's dropped frames next to Jarvis
User decision (2026-11-11): the 95 % GPU cap is off by default because gamemode_tune showed it did not lower nvidia-smi's utilisation and cost 6 to 10 % of the LLM's output. The worry behind it was a video stuttering next to Jarvis, so the new on-demand stage video_drops measures that: - A public Blender upload plays at 1080p60 in a throwaway Chrome/Edge profile (removed afterwards), driven over DevTools with a small standard-library WebSocket client, so no package is added to the already built venvs. - Each window counts the video's dropped frames (the player's own counter) and the gaps over 50/100 ms between shown frames (requestVideoFrameCallback); windows with an ad, a pause, the end or a new video are kept but not counted. - Rows: video alone, Jarvis idle (normal placement with the STT room and 1 GB for other programs), Jarvis busy (LLM answering, STT transcribing ten speakers back to back) without and with the cap in A B B A order, then the video alone again. The cap pauses the engine and the STT worker only, never the browser. - run_m0_video.bat runs just this stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -37,6 +37,7 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
|
||||
| PC 제어 도구 | agent-desktop으로 계산기 버튼을 눌러 12+3=15가 나오는지, 테스트용 브라우저 창(새 임시 프로필)의 입력칸에 글자를 넣고 다시 읽히는지, 디스코드·탐색기 화면을 읽는 시간, 계속 켜 두는 방식(MCP)과 매번 실행하는 방식의 명령 한 번 시간 |
|
||||
| 게임 중 자동 조절 | 이미 받은 Qwen3.6-35B-A3B를 평소 배치(동시 1개)와 게임 중 배치 3개(전문가 층 2GB 맥락 32K, 2GB 맥락 16K, 0GB 맥락 16K)로 차례로 다시 시작해서, 배치마다 전환 시간, 남는 VRAM, 출력 속도, 음성 첫 문장, 맥락 8천 토큰 첫 토큰 시간을 잽니다. 마지막에 평소 배치로 한 번 더 켜서 되돌리는 시간도 잽니다 |
|
||||
| LLM과 음성 인식 함께 | 음성 인식(STT)을 먼저 켜 두고 LLM을 평소 배치(동시 1개, 2개)와 게임 중 배치(전문가 2GB·0GB, 맥락 16K)로 켭니다. LLM을 재는 동안 음성 인식은 10명 동시 전사를 계속 돌립니다. 게임 중 배치는 음성 인식을 GPU에 둔 경우와 CPU에 둔 경우를 둘 다 잽니다. 배치마다 남는 VRAM, LLM 속도, 10명 동시 전사 시간, 한 사람 전사 시간을 적습니다 |
|
||||
| 영상 끊김 (`run_m0_video.bat`로만) | 유튜브 영상을 틀어 두고 Jarvis를 끈 때, 켜 두고 쉴 때, 바쁠 때(GPU 95% 제한 없이와 있을 때)의 끊긴 프레임 수와 긴 멈춤 횟수 |
|
||||
| STT 자리와 부분 전사 간격 | 평소 배치를 STT 자리를 남긴 전문가 층 몫으로 켜고 STT를 쉬지 않고 돌리며 잽니다. 게임 중 배치(전문가 2GB)는 STT를 쉬지 않고 돌릴 때, 부분 전사를 2초마다 할 때, 부분 전사 없이 최종 전사만 할 때를 잽니다(게임은 켜지 않음). 배치마다 남는 VRAM, LLM 속도, 최종 전사 시간과 기다린 시간, STT가 GPU를 쓴 비율을 적습니다 |
|
||||
|
||||
## 개인정보
|
||||
@@ -46,6 +47,7 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
|
||||
- 다른 프로그램의 창 제목은 저장하지 않습니다. 디스코드 분리 창 제목은 저장해도 되는지 물어본 뒤에만 넣습니다.
|
||||
- 게임 감지는 실행 중인 프로그램 이름과 창 모양만 봅니다. 게임 화면은 저장하지 않습니다. 설치된 게임은 이름과 안티치트 유무만 기록합니다.
|
||||
- 음성 테스트는 키트에 들어 있는 합성 음성만 씁니다. 마이크는 쓰지 않습니다.
|
||||
- 영상 끊김 단계(`video_drops`)는 유튜브에 공개된 Blender 영상을 새 임시 브라우저 프로필로 엽니다. 로그인하지 않고, 평소 쓰시는 프로필은 건드리지 않으며, 끝나면 그 임시 프로필을 지웁니다. 저장하는 것은 영상의 프레임 수, 끊긴 프레임 수, 화질, 광고가 나왔는지뿐입니다.
|
||||
- PC 제어 도구 단계는 다른 프로그램의 창 제목과 화면 글자를 저장하지 않습니다. 개수, 걸린 시간, 오류 코드, 계산기 결과 숫자만 저장합니다. 이미 켜 둔 계산기가 있으면 건드리지 않고 그 확인을 건너뛰며, 브라우저는 평소 쓰시는 프로필이 아닌 새 임시 프로필로 엽니다.
|
||||
|
||||
## 옵션 (PowerShell에서 직접 실행할 때)
|
||||
@@ -56,7 +58,7 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
|
||||
.\run_m0.ps1 -Only stt,tts_melo -Redo # 특정 단계만 다시
|
||||
```
|
||||
|
||||
단계 이름: `sysinfo`, `winchecks`, `vdisplay`, `tts_melo`, `tts_cosy2`, `tts_cosy3`, `stt`, `llm_fast`, `llm_recommended`, `tools_fast`, `tools_recommended`, `desktop`, `gamemode`, `gamemode_stt`, `gamemode_tune`, `gamemode_live`(이 단계는 `-Only`로 고를 때만 돕니다)
|
||||
단계 이름: `sysinfo`, `winchecks`, `vdisplay`, `tts_melo`, `tts_cosy2`, `tts_cosy3`, `stt`, `llm_fast`, `llm_recommended`, `tools_fast`, `tools_recommended`, `desktop`, `gamemode`, `gamemode_stt`, `gamemode_tune`, `gamemode_live`, `video_drops`(마지막 두 단계는 `-Only`로 고를 때만 돕니다)
|
||||
|
||||
`tools_fast`, `tools_recommended`는 도구 호출만 다시 잽니다. 처음과 같은 방식(고치기 전)과, 자비스의 수정(지시문 규칙, 계획 단계, 건너뛴 도구 직접 실행)을 넣은 방식을 같은 모델로 비교합니다. 이미 받은 모델을 쓰므로 모델 하나에 몇 분이면 끝납니다.
|
||||
|
||||
@@ -72,4 +74,6 @@ LLM 단계는 8080 포트를 씁니다. 다른 프로그램이 이 포트를 쓰
|
||||
|
||||
`run_m0_game_live.bat`을 더블클릭하면 실제 게임을 켠 상태로 잽니다(`gamemode_live`). 게임을 창 모드로 켜고 플레이 화면까지 들어간 뒤 Enter를 누르고 15초 안에 게임으로 돌아가면 됩니다. 게임 중 배치 4개(전문가 2GB·0GB, 음성 인식 GPU·CPU)를 차례로 재고 10~15분 걸리며, 끝나면 알림음이 울립니다. 게임 화면이나 게임 이름은 저장하지 않고 VRAM 사용량과 속도만 적습니다. 그래픽 메모리를 많이 쓰는 게임이면 배치가 켜지지 않을 수 있는데, 그것도 결과로 적습니다.
|
||||
|
||||
`run_m0_video.bat`을 더블클릭하면 Jarvis 옆에서 보는 영상이 끊기는지 잽니다(`video_drops`). 크롬(없으면 엣지)이 새 임시 프로필로 떠서 유튜브의 Blender 영상(Big Buck Bunny, 1080p 60fps, 소리 끔)을 틉니다. 광고가 나오면 건너뛸 수 있을 때 건너뛰고 끝날 때까지 기다립니다. 영상을 틀어 둔 채로 (1) Jarvis를 끈 상태 1분, (2) LLM(평소 배치, STT 자리와 여유 1GB)과 STT를 켜 두고 쉬는 상태 1분, (3) LLM이 답하고 STT가 10명 동시 전사를 쉬지 않고 하는 상태를 GPU 95% 제한 없이 두 번, 제한을 켜고 두 번, (4) 다시 Jarvis를 끈 상태 1분을 잽니다. 각각 끊긴 프레임 수(유튜브 "전문가용 통계"의 Dropped Frames와 같은 값)와, 화면에 프레임이 50ms·100ms 넘게 안 바뀐 횟수를 셉니다. 10~15분 걸립니다. 그동안 브라우저 창을 가리거나 최소화하지 말고 PC에 손대지 않으시면 됩니다. 인터넷으로 영상을 받으므로 네트워크가 느리면 버퍼링도 따로 적습니다.
|
||||
|
||||
다 끝나고 필요 없으면 작업 폴더(`\JarvisM0`)를 통째로 지우면 됩니다. Python, 모델, Strata(엔진과 설정 포함)는 모두 그 안에 설치하도록 지정해 두었습니다.
|
||||
|
||||
@@ -51,14 +51,18 @@ STAGES = [
|
||||
# STT partial passes while the LLM answers (normal and the 2 GB game placement).
|
||||
("gamemode_tune", "자동 조절: STT 자리와 VRAM 여유 1GB, 부분 전사 간격, GPU 95% 제한", "base", "llm.py",
|
||||
["model=fast", "only=tune"], 0),
|
||||
# Does Jarvis make a YouTube video stutter, and does the 95 % GPU cap help (user decision 2026-11-11:
|
||||
# the cap is off by default until this shows it helps). Opens YouTube, so only when asked for.
|
||||
("video_drops", "영상 끊김: Jarvis를 돌리는 동안 유튜브 영상의 끊긴 프레임", "base", "llm.py",
|
||||
["model=fast", "only=video"], 0),
|
||||
# With the user's game running. Needs someone at the PC, so only when asked for (--only).
|
||||
("gamemode_live", "게임을 켠 상태로 재기 (직접 게임을 켜 주셔야 합니다)", "base", "llm.py",
|
||||
["model=fast", "only=live"], 0),
|
||||
]
|
||||
# Stages left out of a full run: they need the user there after the unattended part.
|
||||
ON_DEMAND = {"gamemode_live"}
|
||||
# Stages left out of a full run: they need the user there after the unattended part, or open a website.
|
||||
ON_DEMAND = {"gamemode_live", "video_drops"}
|
||||
# Stages that start the STT worker next to the LLM and need the stt venv's Python.
|
||||
NEEDS_STT = {"gamemode_stt", "gamemode_tune", "gamemode_live"}
|
||||
NEEDS_STT = {"gamemode_stt", "gamemode_tune", "gamemode_live", "video_drops"}
|
||||
|
||||
|
||||
def free_gb(path: Path) -> float:
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
"""The GPU share cap the app would put on Jarvis (user decision, 2026-11-10: 95 %).
|
||||
"""The GPU share cap the app can put on Jarvis (user decision, 2026-11-10: 95 %; 2026-11-11: off by
|
||||
default, because gamemode_tune showed it did not lower the utilisation; video_drops measures whether
|
||||
it keeps a video from stuttering).
|
||||
|
||||
Jarvis's GPU work runs in two processes: the Strata engine and the voice worker (here the STT
|
||||
load worker). A GPU cannot be told to give one process only part of its time, so the cap pauses
|
||||
|
||||
@@ -13,6 +13,8 @@ Options (--opt key=value):
|
||||
only=tune the normal placement with room left for the STT and 1 GB for other
|
||||
programs, the game placement with the STT doing fewer partial passes
|
||||
while the LLM answers, and both with Jarvis capped at 95 % of the GPU
|
||||
only=video a YouTube video's dropped frames with Jarvis off, idle and busy (the
|
||||
normal placement), busy also with the 95 % GPU cap (video.py)
|
||||
stt_python=PATH the Python of the stt venv (m0.py passes it)
|
||||
"""
|
||||
|
||||
@@ -661,7 +663,7 @@ def measure_external(r: StageResult, url: str, context: int, only: str = "") ->
|
||||
if only == "tools":
|
||||
tools_only(r, c)
|
||||
return
|
||||
if only in LEVEL_SETS:
|
||||
if only in LEVEL_SETS or only == "video":
|
||||
r.note("placements need Strata; not measured against an external server")
|
||||
return
|
||||
with GpuSampler() as g:
|
||||
@@ -680,7 +682,7 @@ def measure_strata(r: StageResult, work: Path, which: str, context: int, only: s
|
||||
return
|
||||
r["plan"] = s.plan(preset, context)
|
||||
r.save()
|
||||
if only in LEVEL_SETS:
|
||||
if only in LEVEL_SETS or only == "video":
|
||||
s.stop()
|
||||
if port_in_use(PORT):
|
||||
r.error("port", f"다른 프로그램이 {PORT} 포트를 쓰고 있습니다. 그 프로그램을 끄고 다시 실행해 주세요.")
|
||||
@@ -688,6 +690,10 @@ def measure_strata(r: StageResult, work: Path, which: str, context: int, only: s
|
||||
if only != "levels" and not stt_python:
|
||||
r.error("stt", "STT 환경을 찾지 못했습니다")
|
||||
return
|
||||
if only == "video":
|
||||
from m0lib import video
|
||||
video.measure(r, s, preset, context, stt_python)
|
||||
return
|
||||
if only == "live":
|
||||
if not ask_for_a_game():
|
||||
r["skipped"] = "no game started"
|
||||
@@ -772,7 +778,7 @@ def main() -> int:
|
||||
context = int(opts.get("context", "32768"))
|
||||
only = opts.get("only", "")
|
||||
name = {"tools": f"tools_{which}", "levels": "gamemode", "together": "gamemode_stt",
|
||||
"live": "gamemode_live", "tune": "gamemode_tune"}.get(only, f"llm_{which}")
|
||||
"live": "gamemode_live", "tune": "gamemode_tune", "video": "video_drops"}.get(only, f"llm_{which}")
|
||||
r = StageResult(name, out)
|
||||
try:
|
||||
if opts.get("url"):
|
||||
|
||||
512
scripts/m0/m0lib/video.py
Normal file
512
scripts/m0/m0lib/video.py
Normal file
@@ -0,0 +1,512 @@
|
||||
"""Stage part: does Jarvis make a video stutter, and does the 95 % GPU cap help?
|
||||
|
||||
User decision (2026-11-11): the cap's default is off, because gamemode_tune showed it did not lower
|
||||
nvidia-smi's utilisation and cost 6 to 10 % of the LLM's output. What the user cared about is a
|
||||
video next to Jarvis, so that is measured directly: a YouTube video plays in a throwaway Chrome/Edge
|
||||
profile (never the user's) and the kit counts its dropped frames and long gaps between shown frames
|
||||
with Jarvis off, Jarvis loaded and idle, and Jarvis busy (the LLM answering, the STT transcribing ten
|
||||
speakers back to back) without and with the cap. The busy rows run in the order A B B A (no cap,
|
||||
cap, cap, no cap) so a slow drift does not count for one side.
|
||||
|
||||
The browser is driven through the Chrome DevTools Protocol with a small standard-library WebSocket
|
||||
client (the kit's venvs are already built, so no package is added). Nothing from the page is saved
|
||||
except the video's own counters, its quality and whether an ad was showing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import struct
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
from m0lib import gpucap, llm
|
||||
from m0lib.common import IS_WINDOWS, StageResult, gpu_memory_mib, say
|
||||
from m0lib.llm_client import ChatClient, http_json
|
||||
|
||||
# Blender's own upload (CC BY 3.0, 10:34, 4K and 1080p at 60 fps): no music claim, rarely ads.
|
||||
VIDEO_URL = "https://www.youtube.com/watch?v=aqz-KE-bpKQ"
|
||||
QUALITY = "hd1080"
|
||||
QUIET_S = 60 # the video alone, and Jarvis loaded but idle
|
||||
SETTLE_S = 5 # after a quality change or a seek, before a window starts
|
||||
MIN_LEFT_S = 240 # a window that could reach the end starts the video again first
|
||||
GAP_MS = (50, 100) # a gap between shown frames this long is a visible hitch at 60 fps
|
||||
ROUNDS = ("busy", "busy_cap95", "busy_cap95_2", "busy_2")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# A minimal WebSocket client (RFC 6455) for one DevTools page.
|
||||
|
||||
def encode_frame(payload: bytes, opcode: int = 1, mask: bytes | None = None) -> bytes:
|
||||
"""One final client frame; a client must mask what it sends."""
|
||||
mask = mask or os.urandom(4)
|
||||
n = len(payload)
|
||||
if n < 126:
|
||||
head = bytes([0x80 | opcode, 0x80 | n])
|
||||
elif n < 1 << 16:
|
||||
head = bytes([0x80 | opcode, 0x80 | 126]) + struct.pack(">H", n)
|
||||
else:
|
||||
head = bytes([0x80 | opcode, 0x80 | 127]) + struct.pack(">Q", n)
|
||||
return head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload))
|
||||
|
||||
|
||||
class Cdp:
|
||||
"""One DevTools WebSocket: ``call`` sends a command and returns its result, skipping events."""
|
||||
|
||||
def __init__(self, ws_url: str, timeout: float = 60, sock: socket.socket | None = None):
|
||||
self._buf = b""
|
||||
self._id = 0
|
||||
if sock is not None: # tests hand in one end of a socket pair, already "upgraded"
|
||||
self.sock = sock
|
||||
return
|
||||
u = urllib.parse.urlparse(ws_url)
|
||||
self.sock = socket.create_connection((u.hostname, u.port), timeout=timeout)
|
||||
key = base64.b64encode(os.urandom(16)).decode()
|
||||
# No Origin header: Chrome only checks --remote-allow-origins when one is sent.
|
||||
self.sock.sendall((f"GET {u.path} HTTP/1.1\r\nHost: {u.hostname}:{u.port}\r\nUpgrade: websocket\r\n"
|
||||
f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n"
|
||||
).encode())
|
||||
while b"\r\n\r\n" not in self._buf:
|
||||
self._more()
|
||||
head, self._buf = self._buf.split(b"\r\n\r\n", 1)
|
||||
if b" 101 " not in head.split(b"\r\n", 1)[0]:
|
||||
raise ConnectionError(f"DevTools refused the WebSocket: {head[:200]!r}")
|
||||
|
||||
def _more(self) -> None:
|
||||
got = self.sock.recv(65536)
|
||||
if not got:
|
||||
raise ConnectionError("DevTools closed the connection")
|
||||
self._buf += got
|
||||
|
||||
def _take(self, n: int) -> bytes:
|
||||
while len(self._buf) < n:
|
||||
self._more()
|
||||
out, self._buf = self._buf[:n], self._buf[n:]
|
||||
return out
|
||||
|
||||
def _frame(self) -> tuple[bool, int, bytes]:
|
||||
b0, b1 = self._take(2)
|
||||
n = b1 & 0x7F
|
||||
if n == 126:
|
||||
n = struct.unpack(">H", self._take(2))[0]
|
||||
elif n == 127:
|
||||
n = struct.unpack(">Q", self._take(8))[0]
|
||||
mask = self._take(4) if b1 & 0x80 else None
|
||||
data = self._take(n)
|
||||
if mask:
|
||||
data = bytes(b ^ mask[i % 4] for i, b in enumerate(data))
|
||||
return bool(b0 & 0x80), b0 & 0x0F, data
|
||||
|
||||
def recv(self) -> str:
|
||||
"""The next whole text message; answers pings, raises on close."""
|
||||
parts: list[bytes] = []
|
||||
while True:
|
||||
fin, op, data = self._frame()
|
||||
if op == 8:
|
||||
raise ConnectionError("DevTools closed the WebSocket")
|
||||
if op == 9:
|
||||
self.sock.sendall(encode_frame(data, opcode=10))
|
||||
continue
|
||||
if op == 10:
|
||||
continue
|
||||
parts.append(data)
|
||||
if fin:
|
||||
return b"".join(parts).decode("utf-8", "replace")
|
||||
|
||||
def call(self, method: str, params: dict | None = None) -> dict:
|
||||
self._id += 1
|
||||
mine = self._id
|
||||
self.sock.sendall(encode_frame(json.dumps({"id": mine, "method": method, "params": params or {}}).encode()))
|
||||
while True:
|
||||
msg = json.loads(self.recv())
|
||||
if msg.get("id") != mine:
|
||||
continue # an event, or the answer to a command that timed out earlier
|
||||
if "error" in msg:
|
||||
raise RuntimeError(f"{method}: {msg['error']}")
|
||||
return msg.get("result") or {}
|
||||
|
||||
def evaluate(self, expression: str) -> Any:
|
||||
res = self.call("Runtime.evaluate", {"expression": expression, "returnByValue": True})
|
||||
if res.get("exceptionDetails"):
|
||||
d = res["exceptionDetails"]
|
||||
raise RuntimeError(f"page script failed: {(d.get('exception') or {}).get('description') or d.get('text')}")
|
||||
return (res.get("result") or {}).get("value")
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self.sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The page: what runs inside YouTube's watch page.
|
||||
|
||||
# Counts shown frames (requestVideoFrameCallback runs once per frame put on screen), the long gaps
|
||||
# between them, buffering and ads; installed again if YouTube swaps the <video> element.
|
||||
_INSTALL = """(() => {
|
||||
const v = document.querySelector('video.html5-main-video') || document.querySelector('video');
|
||||
if (!v || !v.requestVideoFrameCallback) return false;
|
||||
if (window.__m0 && window.__m0.v === v) return true;
|
||||
const m = window.__m0 = {v, shown: 0, last: null, maxGap: 0, gaps: [0, 0], waits: 0, ad: false};
|
||||
v.addEventListener('waiting', () => { m.waits++; });
|
||||
const tick = (now) => {
|
||||
if (m.last !== null) {
|
||||
const g = now - m.last;
|
||||
if (g > m.maxGap) m.maxGap = g;
|
||||
if (g > %(g0)d) m.gaps[0]++;
|
||||
if (g > %(g1)d) m.gaps[1]++;
|
||||
}
|
||||
m.last = now; m.shown++;
|
||||
const p = document.getElementById('movie_player');
|
||||
if (p && p.classList.contains('ad-showing')) m.ad = true;
|
||||
v.requestVideoFrameCallback(tick);
|
||||
};
|
||||
v.requestVideoFrameCallback(tick);
|
||||
return true;
|
||||
})()""" % {"g0": GAP_MS[0], "g1": GAP_MS[1]}
|
||||
|
||||
_RESET = "(() => { const m = window.__m0; m.shown = 0; m.last = null; m.maxGap = 0; m.gaps = [0, 0]; " \
|
||||
"m.waits = 0; m.ad = false; return true; })()"
|
||||
|
||||
_SNAPSHOT = """(() => {
|
||||
const m = window.__m0, v = m.v, p = document.getElementById('movie_player');
|
||||
const q = v.getVideoPlaybackQuality();
|
||||
return {dropped: q.droppedVideoFrames, total: q.totalVideoFrames, shown: m.shown, max_gap_ms: m.maxGap,
|
||||
gaps: m.gaps, waits: m.waits, ad: m.ad || !!(p && p.classList.contains('ad-showing')),
|
||||
paused: v.paused, ended: v.ended, time: v.currentTime, duration: v.duration,
|
||||
width: v.videoWidth, height: v.videoHeight,
|
||||
quality: p && p.getPlaybackQuality ? p.getPlaybackQuality() : null};
|
||||
})()"""
|
||||
|
||||
# One step of getting the video going: skip an ad when it can be skipped, press play.
|
||||
_STEP = """(() => {
|
||||
const p = document.getElementById('movie_player');
|
||||
const v = document.querySelector('video.html5-main-video');
|
||||
if (!p || !v) return {state: 'loading'};
|
||||
if (p.classList.contains('ad-showing')) {
|
||||
const b = document.querySelector('.ytp-skip-ad-button, .ytp-ad-skip-button, .ytp-ad-skip-button-modern');
|
||||
if (b) b.click();
|
||||
return {state: 'ad', skipped: !!b};
|
||||
}
|
||||
if (v.paused && p.playVideo) p.playVideo();
|
||||
return {state: v.readyState >= 3 && !v.paused && v.currentTime > 1 ? 'playing' : 'starting',
|
||||
levels: p.getAvailableQualityLevels ? p.getAvailableQualityLevels() : []};
|
||||
})()"""
|
||||
|
||||
|
||||
def _js(value: Any) -> str:
|
||||
return json.dumps(value)
|
||||
|
||||
|
||||
class VideoPage:
|
||||
"""The watch page in the throwaway browser."""
|
||||
|
||||
def __init__(self, cdp: Cdp, proc: subprocess.Popen | None = None, profile: Path | None = None):
|
||||
self.cdp, self.proc, self.profile = cdp, proc, profile
|
||||
|
||||
def start(self, quality: str = QUALITY, timeout: float = 180) -> dict:
|
||||
"""Wait for the video to play past any ad, then ask for ``quality``. Returns what happened."""
|
||||
out: dict = {"started": False, "ad_seen": False, "ad_skipped": False, "quality_asked": quality}
|
||||
t0 = time.monotonic()
|
||||
while time.monotonic() - t0 < timeout:
|
||||
st = self.cdp.evaluate(_STEP) or {}
|
||||
out["ad_seen"] |= st.get("state") == "ad"
|
||||
out["ad_skipped"] |= bool(st.get("skipped"))
|
||||
if st.get("state") == "playing":
|
||||
out["started"] = True
|
||||
out["levels"] = st.get("levels")
|
||||
break
|
||||
time.sleep(0.5)
|
||||
out["start_s"] = round(time.monotonic() - t0, 1)
|
||||
if not out["started"]:
|
||||
return out
|
||||
if not self.cdp.evaluate(_INSTALL):
|
||||
out["started"] = False
|
||||
out["error"] = "the page has no video that reports its shown frames"
|
||||
return out
|
||||
# The player's own quality setting (Stats for nerds shows the same).
|
||||
self.cdp.evaluate("(() => { const p = document.getElementById('movie_player'); "
|
||||
f"if (p && p.setPlaybackQualityRange) p.setPlaybackQualityRange({_js(quality)}, "
|
||||
f"{_js(quality)}); return true; }})()")
|
||||
t1 = time.monotonic()
|
||||
while time.monotonic() - t1 < 30 and (self.snapshot().get("quality") != quality):
|
||||
time.sleep(1)
|
||||
time.sleep(SETTLE_S)
|
||||
snap = self.snapshot()
|
||||
out.update(quality=snap.get("quality"), width=snap.get("width"), height=snap.get("height"))
|
||||
return out
|
||||
|
||||
def keep_room(self, min_left: float = MIN_LEFT_S) -> None:
|
||||
"""Start the video again when a window could run into its end (YouTube then plays the next one)."""
|
||||
snap = self.snapshot()
|
||||
if (snap.get("duration") or 0) - (snap.get("time") or 0) < min_left:
|
||||
self.cdp.evaluate("(() => { document.getElementById('movie_player').seekTo(5, true); return true; })()")
|
||||
time.sleep(SETTLE_S)
|
||||
self.cdp.evaluate(_INSTALL)
|
||||
|
||||
def reset(self) -> None:
|
||||
self.cdp.evaluate(_RESET)
|
||||
|
||||
def snapshot(self) -> dict:
|
||||
return self.cdp.evaluate(_SNAPSHOT) or {}
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self.cdp.call("Browser.close")
|
||||
except Exception:
|
||||
pass
|
||||
self.cdp.close()
|
||||
if self.proc:
|
||||
try:
|
||||
self.proc.wait(timeout=15)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.proc.kill()
|
||||
if self.profile:
|
||||
shutil.rmtree(self.profile, ignore_errors=True)
|
||||
|
||||
|
||||
def find_browser() -> str | None:
|
||||
if IS_WINDOWS:
|
||||
cands = [os.path.expandvars(p) for p in (
|
||||
r"%ProgramFiles%\Google\Chrome\Application\chrome.exe",
|
||||
r"%LOCALAPPDATA%\Google\Chrome\Application\chrome.exe",
|
||||
r"%ProgramFiles(x86)%\Microsoft\Edge\Application\msedge.exe",
|
||||
r"%ProgramFiles%\Microsoft\Edge\Application\msedge.exe")]
|
||||
return next((p for p in cands if os.path.exists(p)), None)
|
||||
return next((shutil.which(n) for n in ("google-chrome", "chromium", "chromium-browser") if shutil.which(n)), None)
|
||||
|
||||
|
||||
def open_video_page(work: Path, url: str = VIDEO_URL, extra_args: tuple[str, ...] = ()) -> VideoPage | None:
|
||||
"""Open ``url`` in a new throwaway profile with DevTools on a free port. None without a browser."""
|
||||
exe = find_browser()
|
||||
if not exe:
|
||||
return None
|
||||
profile = work / "cache" / "video" / "browser-profile"
|
||||
shutil.rmtree(profile, ignore_errors=True)
|
||||
profile.mkdir(parents=True)
|
||||
# Kept running and drawing even if another window covers part of it; muted (nobody listens).
|
||||
proc = subprocess.Popen([exe, f"--user-data-dir={profile}", "--remote-debugging-port=0", "--no-first-run",
|
||||
"--no-default-browser-check", "--autoplay-policy=no-user-gesture-required",
|
||||
"--mute-audio", "--start-maximized", "--disable-backgrounding-occluded-windows",
|
||||
"--disable-renderer-backgrounding", "--disable-background-timer-throttling",
|
||||
*extra_args, url], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
port_file = profile / "DevToolsActivePort"
|
||||
t0 = time.monotonic()
|
||||
while time.monotonic() - t0 < 60:
|
||||
if port_file.exists() and port_file.read_text(encoding="utf-8").strip():
|
||||
port = int(port_file.read_text(encoding="utf-8").split()[0])
|
||||
for _ in range(30):
|
||||
code, pages = http_json("GET", f"http://127.0.0.1:{port}/json/list", timeout=5)
|
||||
page = next((p for p in (pages or []) if p.get("type") == "page" and "youtube" in p.get("url", "")),
|
||||
None) if code == 200 else None
|
||||
if page:
|
||||
return VideoPage(Cdp(page["webSocketDebuggerUrl"]), proc, profile)
|
||||
time.sleep(1)
|
||||
break
|
||||
if proc.poll() is not None:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
proc.kill()
|
||||
raise RuntimeError("the browser started without DevTools or without the video page")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# One measured window and the comparison.
|
||||
|
||||
def window_stats(before: dict, after: dict, seconds: float) -> dict:
|
||||
"""What the video did between two snapshots (``after`` also carries the shown-frame counters
|
||||
since the reset just before ``before``). A window with an ad, a pause, the end or a reload of the
|
||||
counters is kept but not counted (``valid`` False, with the reason)."""
|
||||
out: dict = {"seconds": round(seconds, 1)}
|
||||
reasons = []
|
||||
if after.get("total", 0) < before.get("total", 0) or after.get("dropped", 0) < before.get("dropped", 0):
|
||||
reasons.append("the video's counters started again (another video was loaded)")
|
||||
for key, why in (("ad", "an ad was showing"), ("paused", "the video was paused"), ("ended", "the video ended")):
|
||||
if after.get(key):
|
||||
reasons.append(why)
|
||||
total = after.get("total", 0) - before.get("total", 0)
|
||||
dropped = after.get("dropped", 0) - before.get("dropped", 0)
|
||||
out.update(frames=total, dropped=dropped, shown=after.get("shown"),
|
||||
dropped_share=round(dropped / total, 4) if total > 0 else None,
|
||||
dropped_per_min=round(dropped / seconds * 60, 1) if seconds > 0 else None,
|
||||
fps=round(total / seconds, 1) if seconds > 0 else None,
|
||||
gaps_over_ms={str(GAP_MS[0]): (after.get("gaps") or [0, 0])[0],
|
||||
str(GAP_MS[1]): (after.get("gaps") or [0, 0])[1]},
|
||||
max_gap_ms=round(after.get("max_gap_ms") or 0, 1), buffering=after.get("waits", 0),
|
||||
quality=after.get("quality"), height=after.get("height"))
|
||||
if total <= 0 and not reasons:
|
||||
reasons.append("the video showed no frames")
|
||||
out["valid"] = not reasons
|
||||
if reasons:
|
||||
out["invalid_because"] = reasons
|
||||
return out
|
||||
|
||||
|
||||
def compare(rows: list[dict]) -> dict:
|
||||
"""The busy rows without and with the cap, added up over the valid windows."""
|
||||
out = {}
|
||||
for name, capped in (("no_cap", False), ("cap95", True)):
|
||||
valid = [x["video"] for x in rows
|
||||
if x.get("jarvis") == "busy" and bool(x.get("gpu_cap")) == capped and (x.get("video") or {}).get("valid")]
|
||||
frames = sum(v["frames"] for v in valid)
|
||||
dropped = sum(v["dropped"] for v in valid)
|
||||
secs = sum(v["seconds"] for v in valid)
|
||||
out[name] = {"windows": len(valid), "frames": frames, "dropped": dropped,
|
||||
"dropped_share": round(dropped / frames, 4) if frames else None,
|
||||
"dropped_per_min": round(dropped / secs * 60, 1) if secs else None,
|
||||
f"gaps_over_{GAP_MS[0]}ms": sum(v["gaps_over_ms"][str(GAP_MS[0])] for v in valid),
|
||||
f"gaps_over_{GAP_MS[1]}ms": sum(v["gaps_over_ms"][str(GAP_MS[1])] for v in valid)}
|
||||
return out
|
||||
|
||||
|
||||
def measure_window(page: VideoPage, work: Callable[[], dict] | None = None,
|
||||
seconds: float | None = None) -> tuple[dict, dict]:
|
||||
"""The video's stats over ``work()`` (or ``seconds`` of nothing), with the GPU's use meanwhile."""
|
||||
page.keep_room()
|
||||
page.reset()
|
||||
before = page.snapshot()
|
||||
t0 = time.perf_counter()
|
||||
done: dict = {}
|
||||
with gpucap.UtilSampler() as util:
|
||||
if work:
|
||||
done = work()
|
||||
else:
|
||||
time.sleep(QUIET_S if seconds is None else seconds)
|
||||
after = page.snapshot()
|
||||
stats = window_stats(before, after, time.perf_counter() - t0)
|
||||
stats["gpu_util"] = util.summary()
|
||||
return stats, done
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The stage.
|
||||
|
||||
ROOM_LEVEL = next(lv for lv in llm.TUNED if lv["key"] == "normal_room_stt_gpu")
|
||||
|
||||
|
||||
def measure(r: StageResult, s: "llm.Strata", preset: dict, context: int, stt_python: str, url: str = VIDEO_URL,
|
||||
quality: str = QUALITY) -> None:
|
||||
"""Video alone, next to Jarvis idle, next to Jarvis busy without and with the cap, alone again."""
|
||||
base = f"http://127.0.0.1:{llm.PORT}/v1"
|
||||
rows: list[dict] = []
|
||||
r["video_rows"] = rows
|
||||
r["video_source"] = {"url": url, "quality_asked": quality}
|
||||
say(" 브라우저(새 임시 프로필)로 유튜브 영상을 틉니다. 끝날 때까지 PC에 손대지 말아 주세요.")
|
||||
try:
|
||||
page = open_video_page(s.log.parent.parent)
|
||||
except Exception as e:
|
||||
r.error("browser", e)
|
||||
return
|
||||
if page is None:
|
||||
r.error("browser", "Chrome이나 Edge를 찾지 못했습니다")
|
||||
return
|
||||
stt = None
|
||||
try:
|
||||
start = page.start(quality)
|
||||
r["video_source"].update(start)
|
||||
if not start["started"]:
|
||||
r.error("video", start.get("error") or "the video did not start playing")
|
||||
return
|
||||
if start.get("quality") != quality:
|
||||
r.note(f"the player stayed at {start.get('quality')} instead of {quality}")
|
||||
|
||||
def row(key: str, jarvis: str, cap: float | None = None) -> dict:
|
||||
x = {"key": key, "jarvis": jarvis, "gpu_cap": cap}
|
||||
used = gpu_memory_mib()
|
||||
if used:
|
||||
x["vram_free_mib"] = used[1] - used[0]
|
||||
rows.append(x)
|
||||
return x
|
||||
|
||||
def keep(x: dict, stats: dict) -> None:
|
||||
x["video"] = stats
|
||||
if not stats["valid"]:
|
||||
r.note(f"{x['key']}: not counted ({'; '.join(stats['invalid_because'])})")
|
||||
v = stats
|
||||
say(f" {x['key']}: 끊긴 프레임 {v['dropped']}/{v['frames']}, {GAP_MS[1]}ms 넘는 멈춤 "
|
||||
f"{v['gaps_over_ms'][str(GAP_MS[1])]}번")
|
||||
r.save()
|
||||
|
||||
say(" - 영상만")
|
||||
x = row("video_only", "off")
|
||||
keep(x, measure_window(page)[0])
|
||||
|
||||
say(" - Jarvis 켜기 (평소 배치, STT 자리와 여유 1GB)")
|
||||
s.stop()
|
||||
stt = llm.SttLoad(stt_python, s.log.parent.parent, "cuda", s.log.with_name("stt_load.log"))
|
||||
load = stt.start()
|
||||
r["stt_load"] = load
|
||||
if load.get("status") != "loaded":
|
||||
r.error("video stt", "; ".join(load.get("errors") or []) or "the STT did not load")
|
||||
return
|
||||
lv = dict(ROOM_LEVEL)
|
||||
place = {"context": lv.get("context", context)}
|
||||
room = llm.stt_room(r, s, preset, place, lv, base)
|
||||
r["stt_room"] = room
|
||||
sets = list(lv["sets"]) + [f"expert-vram={room['expert_vram_gib']}"]
|
||||
r["placement"] = {"sets": sets, "context": place["context"], "parallel": lv["parallel"], "vision": lv["vision"]}
|
||||
launch = s.launch(preset, place["context"], parallel=lv["parallel"], vision=lv["vision"], extra_sets=sets,
|
||||
timeout=3600)
|
||||
if not launch.get("ready"):
|
||||
r.error("video llm", "the normal placement did not start")
|
||||
return
|
||||
c = ChatClient(base)
|
||||
r["server"] = {"model": c.model}
|
||||
c.chat([{"role": "user", "content": "안녕"}], max_tokens=8) # loading is not answer time
|
||||
|
||||
say(" - Jarvis 켜 두고 쉬는 중")
|
||||
x = row("jarvis_idle", "idle")
|
||||
keep(x, measure_window(page)[0])
|
||||
|
||||
stt.go()
|
||||
for key in ROUNDS:
|
||||
cap = llm.GPU_CAP if "cap95" in key else None
|
||||
say(f" - Jarvis가 답하고 STT가 쉬지 않고 전사하는 중{' (GPU 95% 제한)' if cap else ''}")
|
||||
x = row(key, "busy", cap)
|
||||
pids = llm.cap_pids(s, stt) if cap else []
|
||||
if cap and not pids:
|
||||
r.error(f"video {key} gpu cap", "found no engine process to pause")
|
||||
|
||||
def work() -> dict:
|
||||
with gpucap.DutyCap(pids, cap or 1.0) as duty:
|
||||
bench = llm.quick_bench(c)
|
||||
return {"bench": bench, "duty": duty.summary()}
|
||||
|
||||
stats, done = measure_window(page, work)
|
||||
bench = done.get("bench") or {}
|
||||
x["llm"] = {k: bench.get(k) for k in ("decode_tok_s", "voice_first_sentence", "context")}
|
||||
for err in bench.get("errors") or []:
|
||||
r.error(f"video {key}", err)
|
||||
if cap:
|
||||
x["gpu_cap"] = done["duty"]
|
||||
if x["gpu_cap"]["left_paused"]:
|
||||
r.error(f"video {key} gpu cap", f"{x['gpu_cap']['left_paused']} process(es) are still paused")
|
||||
keep(x, stats)
|
||||
busy = stt.finish()
|
||||
r["stt_busy"] = {k: busy.get(k) for k in ("status", "batch10_s", "single_s", "errors", "busy_share")
|
||||
if k in busy}
|
||||
for err in llm.stt_gaps(busy):
|
||||
r.error("video stt", err)
|
||||
stt.close()
|
||||
stt = None
|
||||
s.stop()
|
||||
|
||||
say(" - Jarvis 끄고 영상만 다시")
|
||||
x = row("video_only_again", "off")
|
||||
keep(x, measure_window(page)[0])
|
||||
r["comparison"] = compare(rows)
|
||||
except Exception as e:
|
||||
r.error("video", e)
|
||||
finally:
|
||||
if stt:
|
||||
stt.close()
|
||||
s.stop()
|
||||
page.close()
|
||||
5
scripts/m0/run_m0_video.bat
Normal file
5
scripts/m0/run_m0_video.bat
Normal file
@@ -0,0 +1,5 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0run_m0.ps1" -Only video_drops -Redo %*
|
||||
echo.
|
||||
pause
|
||||
@@ -1206,3 +1206,208 @@ def test_a_suspend_or_resume_that_raises_does_not_end_the_cap_with_processes_pau
|
||||
with gpucap.DutyCap([1], 0.9, period=0.02, suspend=procs.suspend, resume=resume) as d:
|
||||
_time.sleep(0.2)
|
||||
assert procs.count[1] == 0 and d.summary()["left_paused"] == 0
|
||||
|
||||
|
||||
# --- video_drops: a YouTube video's dropped frames next to Jarvis, with and without the cap --
|
||||
|
||||
|
||||
def _server_frame(payload: bytes, opcode: int = 1, fin: bool = True) -> bytes:
|
||||
"""A frame as Chrome sends it (servers do not mask)."""
|
||||
import struct
|
||||
n = len(payload)
|
||||
head = bytes([(0x80 if fin else 0) | opcode])
|
||||
if n < 126:
|
||||
head += bytes([n])
|
||||
elif n < 1 << 16:
|
||||
head += bytes([126]) + struct.pack(">H", n)
|
||||
else:
|
||||
head += bytes([127]) + struct.pack(">Q", n)
|
||||
return head + payload
|
||||
|
||||
|
||||
def _read_client_frame(sock) -> tuple[int, bytes]:
|
||||
"""Unmask one frame the client sent (clients must mask)."""
|
||||
import struct
|
||||
b0, b1 = sock.recv(2)
|
||||
n = b1 & 0x7F
|
||||
if n == 126:
|
||||
n = struct.unpack(">H", sock.recv(2))[0]
|
||||
elif n == 127:
|
||||
n = struct.unpack(">Q", sock.recv(8))[0]
|
||||
assert b1 & 0x80, "a client frame must be masked"
|
||||
mask = sock.recv(4)
|
||||
data = b""
|
||||
while len(data) < n:
|
||||
data += sock.recv(n - len(data))
|
||||
return b0 & 0x0F, bytes(b ^ mask[i % 4] for i, b in enumerate(data))
|
||||
|
||||
|
||||
def test_devtools_call_returns_its_own_answer_across_events_pings_and_fragments():
|
||||
import socket
|
||||
from m0lib import video
|
||||
ours, chrome = socket.socketpair()
|
||||
cdp = video.Cdp("", sock=ours)
|
||||
big = json.dumps({"id": 1, "result": {"result": {"value": "x" * 70000}}}).encode()
|
||||
chrome.sendall(_server_frame(json.dumps({"method": "Page.frameNavigated"}).encode()) # an event
|
||||
+ _server_frame(b"are you there", opcode=9) # a ping
|
||||
+ _server_frame(big[:300], fin=False) + _server_frame(big[300:], opcode=0))
|
||||
|
||||
got = {}
|
||||
t = threading.Thread(target=lambda: got.update(v=cdp.evaluate("1")))
|
||||
t.start()
|
||||
op, sent = _read_client_frame(chrome)
|
||||
assert op == 1 and json.loads(sent)["method"] == "Runtime.evaluate"
|
||||
op, pong = _read_client_frame(chrome)
|
||||
assert (op, pong) == (10, b"are you there")
|
||||
t.join(timeout=10)
|
||||
assert got["v"] == "x" * 70000
|
||||
cdp.close()
|
||||
chrome.close()
|
||||
|
||||
|
||||
def test_devtools_errors_and_a_closed_socket_are_raised_not_hidden():
|
||||
import socket
|
||||
from m0lib import video
|
||||
ours, chrome = socket.socketpair()
|
||||
cdp = video.Cdp("", sock=ours)
|
||||
chrome.sendall(_server_frame(json.dumps({"id": 1, "error": {"message": "no such method"}}).encode()))
|
||||
with pytest.raises(RuntimeError, match="no such method"):
|
||||
cdp.call("Nope.nothing")
|
||||
chrome.sendall(_server_frame(json.dumps({"id": 2, "result": {
|
||||
"exceptionDetails": {"text": "Uncaught", "exception": {"description": "TypeError: m is undefined"}}}}).encode()))
|
||||
with pytest.raises(RuntimeError, match="m is undefined"):
|
||||
cdp.evaluate("m.v")
|
||||
chrome.sendall(_server_frame(b"", opcode=8))
|
||||
with pytest.raises(ConnectionError):
|
||||
cdp.call("Runtime.evaluate")
|
||||
cdp.close()
|
||||
chrome.close()
|
||||
|
||||
|
||||
def _snap(total, dropped, **kw):
|
||||
return {"total": total, "dropped": dropped, "shown": kw.pop("shown", total), "gaps": kw.pop("gaps", [0, 0]),
|
||||
"max_gap_ms": kw.pop("max_gap_ms", 17.0), "waits": 0, "quality": "hd1080", "height": 1080, **kw}
|
||||
|
||||
|
||||
def test_a_window_counts_the_frames_the_video_dropped_meanwhile():
|
||||
from m0lib import video
|
||||
w = video.window_stats(_snap(1000, 5), _snap(4600, 23, gaps=[4, 1], max_gap_ms=140.0), 60.0)
|
||||
assert w["valid"] and w["frames"] == 3600 and w["dropped"] == 18
|
||||
assert w["fps"] == 60.0 and w["dropped_per_min"] == 18.0 and w["dropped_share"] == 0.005
|
||||
assert w["gaps_over_ms"] == {str(video.GAP_MS[0]): 4, str(video.GAP_MS[1]): 1} and w["max_gap_ms"] == 140.0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("after", [_snap(4600, 9, ad=True), _snap(4600, 9, paused=True), _snap(4600, 9, ended=True),
|
||||
_snap(300, 0), _snap(1000, 5)])
|
||||
def test_a_window_with_an_ad_a_pause_the_end_a_new_video_or_no_frames_is_kept_but_not_counted(after):
|
||||
from m0lib import video
|
||||
w = video.window_stats(_snap(1000, 5), after, 60.0)
|
||||
assert not w["valid"] and w["invalid_because"]
|
||||
|
||||
|
||||
def test_the_comparison_adds_up_only_valid_busy_windows_on_each_side_of_the_cap():
|
||||
from m0lib import video
|
||||
|
||||
def row(jarvis, cap, frames, dropped, valid=True):
|
||||
return {"jarvis": jarvis, "gpu_cap": {"cap": 0.95} if cap else None,
|
||||
"video": {"valid": valid, "frames": frames, "dropped": dropped, "seconds": 60.0,
|
||||
"gaps_over_ms": {str(g): dropped for g in video.GAP_MS}}}
|
||||
|
||||
rows = [row("off", False, 3600, 0), row("busy", False, 3600, 36), row("busy", True, 3600, 6),
|
||||
row("busy", True, 3600, 1000, valid=False), row("busy", False, 3600, 18)]
|
||||
c = video.compare(rows)
|
||||
assert c["no_cap"]["windows"] == 2 and c["no_cap"]["dropped"] == 54 and c["no_cap"]["dropped_per_min"] == 27.0
|
||||
assert c["cap95"]["windows"] == 1 and c["cap95"]["dropped"] == 6 and c["cap95"]["dropped_share"] == round(6 / 3600, 4)
|
||||
|
||||
|
||||
class _FakePage:
|
||||
"""The watch page: 60 frames shown per snapshot step, a few dropped; records what the stage did."""
|
||||
|
||||
def __init__(self, starts=True):
|
||||
self.starts = starts
|
||||
self.total = self.dropped = 0
|
||||
self.events: list[str] = []
|
||||
self.closed = False
|
||||
|
||||
def start(self, quality="hd1080", timeout=180):
|
||||
self.events.append("video_start")
|
||||
return {"started": self.starts, "quality": quality if self.starts else None, "ad_seen": False}
|
||||
|
||||
def keep_room(self):
|
||||
pass
|
||||
|
||||
def reset(self):
|
||||
pass
|
||||
|
||||
def snapshot(self):
|
||||
self.total += 3600
|
||||
self.dropped += 2
|
||||
return _snap(self.total, self.dropped)
|
||||
|
||||
def close(self):
|
||||
self.closed = True
|
||||
|
||||
|
||||
def _run_video_stage(tmp_path, monkeypatch, fake_server, page, stt=None):
|
||||
from m0lib import video
|
||||
monkeypatch.setattr(video, "QUIET_S", 0.01)
|
||||
monkeypatch.setattr(video, "open_video_page", lambda work, *a, **k: page)
|
||||
stt = _FakeStt() if stt is None else stt
|
||||
r, started, running = _run_levels_stage(tmp_path, monkeypatch, fake_server, _vram, only="video",
|
||||
extra_opts=["stt_python=py"], stt=stt)
|
||||
return r, started, running, stt
|
||||
|
||||
|
||||
def test_video_stage_compares_the_video_alone_jarvis_idle_and_busy_with_and_without_the_cap(
|
||||
tmp_path, monkeypatch, fake_server):
|
||||
from m0lib import video
|
||||
page = _FakePage()
|
||||
r, started, running, stt = _run_video_stage(tmp_path, monkeypatch, fake_server, page)
|
||||
assert r["stage"] == "video_drops" and r["status"] == "ok"
|
||||
rows = r["video_rows"]
|
||||
# Alone first and last; idle before busy; the busy rows alternate A B B A around the cap.
|
||||
assert [x["jarvis"] for x in rows] == ["off", "idle"] + ["busy"] * len(video.ROUNDS) + ["off"]
|
||||
busy = [x for x in rows if x["jarvis"] == "busy"]
|
||||
capped = [bool(x["gpu_cap"]) for x in busy]
|
||||
assert capped == [False, True, True, False]
|
||||
assert all(x["video"]["valid"] and x["video"]["frames"] > 0 for x in rows)
|
||||
for x in busy:
|
||||
assert x["llm"]["decode_tok_s"]["n"] > 0 # the LLM answered during the window
|
||||
if x["gpu_cap"]:
|
||||
assert x["gpu_cap"]["cap"] == llm.GPU_CAP and x["gpu_cap"]["cycles"] > 0
|
||||
assert x["gpu_cap"]["left_paused"] == 0
|
||||
# Jarvis is the normal placement: the STT room probe, then one start with the experts it leaves.
|
||||
assert len(started) == 2 and _expert_vram(started[0]) == 0 and _expert_vram(started[1]) > 0
|
||||
assert r["placement"]["sets"] == started[1]["sets"]
|
||||
# The STT is loaded before the LLM and transcribes only once Jarvis is busy.
|
||||
assert stt.events[:4] == ["stt_loaded", "llm_start", "llm_start", "stt_go"]
|
||||
assert set(r["comparison"]) == {"no_cap", "cap95"} and r["comparison"]["cap95"]["windows"] == 2
|
||||
# Nothing is left behind: the browser closed, the model stopped, the STT gone, nothing paused.
|
||||
assert page.closed and running["now"] is None and stt.loaded is None and not running["held"]
|
||||
|
||||
|
||||
def test_the_cap_in_the_video_stage_pauses_jarvis_and_never_the_browser(tmp_path, monkeypatch, fake_server):
|
||||
paused = set()
|
||||
from m0lib import video
|
||||
page = _FakePage()
|
||||
monkeypatch.setattr(video, "QUIET_S", 0.01)
|
||||
monkeypatch.setattr(video, "open_video_page", lambda work, *a, **k: page)
|
||||
_run_levels_stage(tmp_path, monkeypatch, fake_server, _vram, only="video", extra_opts=["stt_python=py"],
|
||||
stt=_FakeStt(), paused=paused)
|
||||
browser = next(p["pid"] for p in _procs(tmp_path / "strata") if "chrome" in p["exe"])
|
||||
assert paused == {20, 21, STT_LAUNCHER_PID, STT_LAUNCHER_PID + 1} and browser not in paused
|
||||
|
||||
|
||||
def test_video_stage_without_a_browser_or_a_playing_video_starts_no_jarvis(tmp_path, monkeypatch, fake_server):
|
||||
r, started, _, stt = _run_video_stage(tmp_path, monkeypatch, fake_server, None)
|
||||
assert r["status"] == "failed" and not started and not stt.events
|
||||
page = _FakePage(starts=False)
|
||||
r, started, _, stt = _run_video_stage(tmp_path, monkeypatch, fake_server, page)
|
||||
assert r["status"] == "failed" and not started and not stt.events and page.closed
|
||||
|
||||
|
||||
def test_video_stage_is_on_demand_with_the_stt_python_and_a_one_click_runner():
|
||||
import m0
|
||||
assert "video_drops" in m0.ON_DEMAND and "video_drops" in m0.NEEDS_STT
|
||||
bat = (Path(m0.KIT) / "run_m0_video.bat").read_text(encoding="utf-8")
|
||||
assert "-Only video_drops" in bat and "-Redo" in bat
|
||||
|
||||
Reference in New Issue
Block a user