fix: recordings keep only what was played, newest waiting request kept, video pages opened play

- Playback tells the worker how much of each request was played (tts_played); the worker keeps a
  spoken answer's recording only that far (an answer stopped mid-way no longer keeps its unheard
  rest, and is marked stopped); nothing played or never reported: no file.
- Old recording day folders are also removed with the first recording of each new day.
- response.interrupt off, latest: a request whose final comes after a newer one's is replaced
  instead of replacing it.
- botBrowser presses play on a page's paused main player also after open, not only after a click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-16 15:56:40 +09:00
parent fdcaae4820
commit ec2672cd4a
12 changed files with 176 additions and 33 deletions

View File

@@ -224,6 +224,7 @@ class BrowserSession:
raise BrowserError(f"the page did not open ({r['errorText']})")
self._settle(started=True)
self._wait_media()
self._press_play()
return self.read()
def click(self, target: str) -> Dict[str, Any]:
@@ -232,15 +233,19 @@ class BrowserSession:
raise BrowserError("no link or button with that label")
self._settle()
self._wait_media()
# Clicked through to a video page whose player did not start: look again and press play.
if self._eval("location.href") != before:
self._press_play()
return self.read()
def _press_play(self) -> None:
"""On a page just reached whose main player did not start: look again and press play."""
try:
if self._eval("location.href") != before and self._eval(_PLAY, gesture=True):
if self._eval(_PLAY, gesture=True):
debug_log("bot browser: pressed play on the page's paused player", "browser")
self._wait_media()
except BrowserError as e:
if "too long" in str(e):
raise
return self.read()
def type(self, text: str, field: Optional[str] = None, submit: bool = False) -> Dict[str, Any]:
if not self._eval(_FOCUS % json.dumps(field or "")):

View File

@@ -54,10 +54,12 @@ when muted; no line without any), text, and the numbered links and buttons, all
After `open` and `click`, a page whose sound or video has not started yet (none plays and one is at
0:00) gets up to 3 s to start before it is read, so the answer says truly whether it plays (a
video page's player starts a moment after the page loads; the bot browser lets pages start sound,
admin.spec.md, "Bot browser"). When a `click` led to another address and the page's main player (a
video at least half the window wide, with a source) is still paused at 0:00 after that wait, the tool
looks again and starts it (`play()` with a user gesture, as the user's own click on the video would
have), then waits for it once more; `open` and `read` leave a page's player as the page has it.
admin.spec.md, "Bot browser"). When an `open` (or a `read` with an address), or a `click` that led to
another address, reached a page whose main player (a video at least half the window wide, with a
source) is still paused at 0:00 after that wait, the tool looks again and starts it (`play()` with a
user gesture, as the user's own click on the video would have), then waits for it once more; the
Media line then says whether it truly plays. Sound or video that is not the main player (an audio
element, a small video) and a click that stays on the same page leave it as the page has it.
(Live, 2026-11-16: a video page clicked for "play X" stayed paused behind an ad and the answer said
it played.) Failures come back as `Error: ...` with `success=False`.

View File

@@ -3,7 +3,7 @@
Off unless the app sets JARVIS_RECORDINGS_DIR (setting `voice.record`). Each finished utterance and
each spoken answer becomes a WAV file in a folder per local day, with one JSON line per file in that
day's `index.jsonl` (time, who, text). Day folders older than KEEP_DAYS are removed when the worker
starts. Writing happens on a thread of its own and a failure never stops the voice path.
starts and with the first recording of each new day. Writing happens on a thread of its own and a failure never stops the voice path.
"""
from __future__ import annotations
@@ -36,7 +36,9 @@ class Recorder:
self.keep_days = keep_days
self.root.mkdir(parents=True, exist_ok=True)
self._pool = ThreadPoolExecutor(max_workers=1, thread_name_prefix="recordings")
self.purge(time.time() if now is None else now)
now = time.time() if now is None else now
self._purged_day = date.fromtimestamp(now)
self.purge(now)
def purge(self, now: float) -> int:
"""Remove day folders older than keep_days (by their name, the local day)."""
@@ -58,7 +60,7 @@ class Recorder:
{"kind": "said", "who": speaker, "name": name, "channel": channel, "text": text})
def spoke(self, ts: float, rid: str, text: str, pcm: np.ndarray, rate: int, cancelled: bool) -> None:
"""A spoken answer: int16 mono samples at `rate`, as far as it was made."""
"""A spoken answer: int16 mono samples at `rate`, as far as the app played it."""
self._submit(ts, "jarvis", np.asarray(pcm, dtype="<i2"), rate,
{"kind": "spoke", "who": "jarvis", "id": rid, "text": text, "cancelled": cancelled})
@@ -73,6 +75,9 @@ class Recorder:
def _write(self, ts: float, tag: str, pcm: np.ndarray, rate: int, entry: dict) -> None:
try:
if date.fromtimestamp(ts) != self._purged_day: # a new day while running
self._purged_day = date.fromtimestamp(ts)
self.purge(ts)
lt = time.localtime(ts)
day = self.root / time.strftime("%Y-%m-%d", lt)
day.mkdir(parents=True, exist_ok=True)

View File

@@ -246,7 +246,8 @@ class Session:
self.names: dict[str, str] = {}
self.speakers: dict[str, _Speaker] = {}
self.tts_requests: dict[str, TtsRequest] = {}
# With recordings on: each spoken answer's text and mono samples until it is done.
# With recordings on: each spoken answer's text and mono samples, kept until it is made and the
# app said how much of it was played (`tts_played`).
self.tts_recorded: dict[str, dict] = {}
# Spoken replies whose speech was cancelled while the brain was still answering.
self.cancelled_replies: set[str] = set()
@@ -390,7 +391,7 @@ class Session:
kind = msg.get("type")
handler = {"speaking_end": self._on_speaking_end, "speaker_leave": self._on_speaker_leave,
"config": self._on_config, "tts": self._on_tts, "tts_cancel": self._on_tts_cancel,
"tts_speed": self._on_tts_speed,
"tts_speed": self._on_tts_speed, "tts_played": self._on_tts_played,
"voice_join": self._on_voice_join, "voice_leave": lambda m: self._leave_voice(),
"member_join": self._on_member_join, "member_leave": self._on_member_leave,
"chat_message": self._on_chat_message, "chat_edit": self._on_chat_edit,
@@ -462,8 +463,11 @@ class Session:
self.error("bad_message", "this tts id is already closed", id=rid)
return
if self.worker.recorder is not None:
rec = self.tts_recorded.setdefault(rid, {"text": "", "pcm": [], "at": None})
rec = self.tts_recorded.setdefault(rid, {"text": "", "pcm": [], "at": None, "rate": req.rate,
"made": False, "played": None})
rec["text"] += text
while len(self.tts_recorded) > 50: # never reported as played: not kept forever
self.tts_recorded.pop(next(iter(self.tts_recorded)))
tts.feed(req, text, more)
def _on_tts_cancel(self, msg: dict) -> None:
@@ -959,13 +963,34 @@ class Session:
def _tts_done(self, req: TtsRequest) -> None:
self.tts_requests.pop(req.id, None)
rec = self.tts_recorded.pop(req.id, None)
if rec is not None and rec["pcm"] and self.worker.recorder is not None:
self.worker.recorder.spoke(rec["at"], req.id, rec["text"], np.concatenate(rec["pcm"]), req.rate,
req.cancelled or req.failed)
rec = self.tts_recorded.get(req.id)
if rec is not None:
rec["made"] = True
self._record_spoken(req.id)
self.send({"type": "tts_done", "id": req.id, "chunks": req.chunks, "cancelled": req.cancelled,
"failed": req.failed})
self._delivered(req.id, not (req.cancelled or req.failed))
def _on_tts_played(self, msg: dict) -> None:
"""How much of a spoken answer the app played: the recording keeps that much."""
rec = self.tts_recorded.get(msg.get("id"))
played = msg.get("played")
if rec is None or isinstance(played, bool) or not isinstance(played, int) or played < 0:
return
rec["played"], rec["cancelled"] = played, bool(msg.get("cancelled"))
self._record_spoken(msg["id"])
def _record_spoken(self, rid: str) -> None:
"""Keep a spoken answer once it is made and the app said how much of it was heard (whole
sentences, the last one maybe cut off by a stop)."""
rec = self.tts_recorded.get(rid)
if rec is None or not rec["made"] or rec["played"] is None:
return
self.tts_recorded.pop(rid)
heard = rec["pcm"][:rec["played"]]
if heard and self.worker.recorder is not None:
self.worker.recorder.spoke(rec["at"], rid, rec["text"], np.concatenate(heard), rec["rate"],
rec["cancelled"])
def _tts_error(self, req: TtsRequest, message: str) -> None:
self.error("tts_failed", message, id=req.id)

View File

@@ -165,6 +165,7 @@ line (-0.6) would lose more real words than it saves.
-> {"type": "tts", "id": "r1", "text": "...", "more": false} // more=true: more text follows for r1
-> {"type": "tts_cancel", "id": "r1"}
-> {"type": "tts_speed", "speed": 1.8} // 0.5-2.0, from the next sentence on
-> {"type": "tts_played", "id": "r1", "played": 1, "cancelled": true} // how much the app played (Recordings)
<- binary 0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}
```
@@ -481,15 +482,20 @@ sets `JARVIS_RECORDINGS_DIR` (setting `voice.record`, off by default; the app pa
- each finished utterance (every final, also one with no text) as a 16 kHz mono WAV of the clip as it
was cut for transcription, before loudness levelling: `<day>/<HHMMSS>-<ms>-said-<speaker>.wav`;
- each spoken answer (every `tts` request that made sound) as a mono WAV at the session's TTS rate,
as far as it was made (also when cancelled or failed): `<day>/<HHMMSS>-<ms>-jarvis.wav`, timed at
its first sentence;
- each spoken answer as a mono WAV at the session's TTS rate, as far as the app played it:
`<day>/<HHMMSS>-<ms>-jarvis.wav`, timed at its first sentence. The app sends `tts_played` (`played`:
how many of its sentences started playing, `cancelled`: whether it was stopped) when each request
ends; the worker keeps the request's sentences until it is made and that report came, then writes
the first `played` sentences (the last one maybe cut off by the stop; it may have been made in full)
with `cancelled`. Nothing played (0), never reported (the session ends, an older app; at most 50
are held) or ids it does not know (the earcon): no file;
- one JSON line per file in `<day>/index.jsonl`: `{file, at, ms, kind: said|spoke, who, name,
channel, text, id, cancelled}` (the transcript, or the text sent to `tts`).
`<day>` is the local day. Files are written on a thread of their own, so the voice path never waits,
and a failed write is a debug line (no text) that changes nothing else. Day folders older than 7 days
are removed when the worker starts. Nothing is recorded without the variable.
are removed when the worker starts and with the first recording of each new day while it runs.
Nothing is recorded without the variable.
### Errors