fix: recordings keep only what was played, newest waiting request kept, video pages opened play
- Playback tells the worker how much of each request was played (tts_played); the worker keeps a spoken answer's recording only that far (an answer stopped mid-way no longer keeps its unheard rest, and is marked stopped); nothing played or never reported: no file. - Old recording day folders are also removed with the first recording of each new day. - response.interrupt off, latest: a request whose final comes after a newer one's is replaced instead of replacing it. - botBrowser presses play on a page's paused main player also after open, not only after a click. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -224,6 +224,7 @@ class BrowserSession:
|
||||
raise BrowserError(f"the page did not open ({r['errorText']})")
|
||||
self._settle(started=True)
|
||||
self._wait_media()
|
||||
self._press_play()
|
||||
return self.read()
|
||||
|
||||
def click(self, target: str) -> Dict[str, Any]:
|
||||
@@ -232,15 +233,19 @@ class BrowserSession:
|
||||
raise BrowserError("no link or button with that label")
|
||||
self._settle()
|
||||
self._wait_media()
|
||||
# Clicked through to a video page whose player did not start: look again and press play.
|
||||
if self._eval("location.href") != before:
|
||||
self._press_play()
|
||||
return self.read()
|
||||
|
||||
def _press_play(self) -> None:
|
||||
"""On a page just reached whose main player did not start: look again and press play."""
|
||||
try:
|
||||
if self._eval("location.href") != before and self._eval(_PLAY, gesture=True):
|
||||
if self._eval(_PLAY, gesture=True):
|
||||
debug_log("bot browser: pressed play on the page's paused player", "browser")
|
||||
self._wait_media()
|
||||
except BrowserError as e:
|
||||
if "too long" in str(e):
|
||||
raise
|
||||
return self.read()
|
||||
|
||||
def type(self, text: str, field: Optional[str] = None, submit: bool = False) -> Dict[str, Any]:
|
||||
if not self._eval(_FOCUS % json.dumps(field or "")):
|
||||
|
||||
@@ -54,10 +54,12 @@ when muted; no line without any), text, and the numbered links and buttons, all
|
||||
After `open` and `click`, a page whose sound or video has not started yet (none plays and one is at
|
||||
0:00) gets up to 3 s to start before it is read, so the answer says truly whether it plays (a
|
||||
video page's player starts a moment after the page loads; the bot browser lets pages start sound,
|
||||
admin.spec.md, "Bot browser"). When a `click` led to another address and the page's main player (a
|
||||
video at least half the window wide, with a source) is still paused at 0:00 after that wait, the tool
|
||||
looks again and starts it (`play()` with a user gesture, as the user's own click on the video would
|
||||
have), then waits for it once more; `open` and `read` leave a page's player as the page has it.
|
||||
admin.spec.md, "Bot browser"). When an `open` (or a `read` with an address), or a `click` that led to
|
||||
another address, reached a page whose main player (a video at least half the window wide, with a
|
||||
source) is still paused at 0:00 after that wait, the tool looks again and starts it (`play()` with a
|
||||
user gesture, as the user's own click on the video would have), then waits for it once more; the
|
||||
Media line then says whether it truly plays. Sound or video that is not the main player (an audio
|
||||
element, a small video) and a click that stays on the same page leave it as the page has it.
|
||||
(Live, 2026-11-16: a video page clicked for "play X" stayed paused behind an ad and the answer said
|
||||
it played.) Failures come back as `Error: ...` with `success=False`.
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Off unless the app sets JARVIS_RECORDINGS_DIR (setting `voice.record`). Each finished utterance and
|
||||
each spoken answer becomes a WAV file in a folder per local day, with one JSON line per file in that
|
||||
day's `index.jsonl` (time, who, text). Day folders older than KEEP_DAYS are removed when the worker
|
||||
starts. Writing happens on a thread of its own and a failure never stops the voice path.
|
||||
starts and with the first recording of each new day. Writing happens on a thread of its own and a failure never stops the voice path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -36,7 +36,9 @@ class Recorder:
|
||||
self.keep_days = keep_days
|
||||
self.root.mkdir(parents=True, exist_ok=True)
|
||||
self._pool = ThreadPoolExecutor(max_workers=1, thread_name_prefix="recordings")
|
||||
self.purge(time.time() if now is None else now)
|
||||
now = time.time() if now is None else now
|
||||
self._purged_day = date.fromtimestamp(now)
|
||||
self.purge(now)
|
||||
|
||||
def purge(self, now: float) -> int:
|
||||
"""Remove day folders older than keep_days (by their name, the local day)."""
|
||||
@@ -58,7 +60,7 @@ class Recorder:
|
||||
{"kind": "said", "who": speaker, "name": name, "channel": channel, "text": text})
|
||||
|
||||
def spoke(self, ts: float, rid: str, text: str, pcm: np.ndarray, rate: int, cancelled: bool) -> None:
|
||||
"""A spoken answer: int16 mono samples at `rate`, as far as it was made."""
|
||||
"""A spoken answer: int16 mono samples at `rate`, as far as the app played it."""
|
||||
self._submit(ts, "jarvis", np.asarray(pcm, dtype="<i2"), rate,
|
||||
{"kind": "spoke", "who": "jarvis", "id": rid, "text": text, "cancelled": cancelled})
|
||||
|
||||
@@ -73,6 +75,9 @@ class Recorder:
|
||||
|
||||
def _write(self, ts: float, tag: str, pcm: np.ndarray, rate: int, entry: dict) -> None:
|
||||
try:
|
||||
if date.fromtimestamp(ts) != self._purged_day: # a new day while running
|
||||
self._purged_day = date.fromtimestamp(ts)
|
||||
self.purge(ts)
|
||||
lt = time.localtime(ts)
|
||||
day = self.root / time.strftime("%Y-%m-%d", lt)
|
||||
day.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -246,7 +246,8 @@ class Session:
|
||||
self.names: dict[str, str] = {}
|
||||
self.speakers: dict[str, _Speaker] = {}
|
||||
self.tts_requests: dict[str, TtsRequest] = {}
|
||||
# With recordings on: each spoken answer's text and mono samples until it is done.
|
||||
# With recordings on: each spoken answer's text and mono samples, kept until it is made and the
|
||||
# app said how much of it was played (`tts_played`).
|
||||
self.tts_recorded: dict[str, dict] = {}
|
||||
# Spoken replies whose speech was cancelled while the brain was still answering.
|
||||
self.cancelled_replies: set[str] = set()
|
||||
@@ -390,7 +391,7 @@ class Session:
|
||||
kind = msg.get("type")
|
||||
handler = {"speaking_end": self._on_speaking_end, "speaker_leave": self._on_speaker_leave,
|
||||
"config": self._on_config, "tts": self._on_tts, "tts_cancel": self._on_tts_cancel,
|
||||
"tts_speed": self._on_tts_speed,
|
||||
"tts_speed": self._on_tts_speed, "tts_played": self._on_tts_played,
|
||||
"voice_join": self._on_voice_join, "voice_leave": lambda m: self._leave_voice(),
|
||||
"member_join": self._on_member_join, "member_leave": self._on_member_leave,
|
||||
"chat_message": self._on_chat_message, "chat_edit": self._on_chat_edit,
|
||||
@@ -462,8 +463,11 @@ class Session:
|
||||
self.error("bad_message", "this tts id is already closed", id=rid)
|
||||
return
|
||||
if self.worker.recorder is not None:
|
||||
rec = self.tts_recorded.setdefault(rid, {"text": "", "pcm": [], "at": None})
|
||||
rec = self.tts_recorded.setdefault(rid, {"text": "", "pcm": [], "at": None, "rate": req.rate,
|
||||
"made": False, "played": None})
|
||||
rec["text"] += text
|
||||
while len(self.tts_recorded) > 50: # never reported as played: not kept forever
|
||||
self.tts_recorded.pop(next(iter(self.tts_recorded)))
|
||||
tts.feed(req, text, more)
|
||||
|
||||
def _on_tts_cancel(self, msg: dict) -> None:
|
||||
@@ -959,13 +963,34 @@ class Session:
|
||||
|
||||
def _tts_done(self, req: TtsRequest) -> None:
|
||||
self.tts_requests.pop(req.id, None)
|
||||
rec = self.tts_recorded.pop(req.id, None)
|
||||
if rec is not None and rec["pcm"] and self.worker.recorder is not None:
|
||||
self.worker.recorder.spoke(rec["at"], req.id, rec["text"], np.concatenate(rec["pcm"]), req.rate,
|
||||
req.cancelled or req.failed)
|
||||
rec = self.tts_recorded.get(req.id)
|
||||
if rec is not None:
|
||||
rec["made"] = True
|
||||
self._record_spoken(req.id)
|
||||
self.send({"type": "tts_done", "id": req.id, "chunks": req.chunks, "cancelled": req.cancelled,
|
||||
"failed": req.failed})
|
||||
self._delivered(req.id, not (req.cancelled or req.failed))
|
||||
|
||||
def _on_tts_played(self, msg: dict) -> None:
|
||||
"""How much of a spoken answer the app played: the recording keeps that much."""
|
||||
rec = self.tts_recorded.get(msg.get("id"))
|
||||
played = msg.get("played")
|
||||
if rec is None or isinstance(played, bool) or not isinstance(played, int) or played < 0:
|
||||
return
|
||||
rec["played"], rec["cancelled"] = played, bool(msg.get("cancelled"))
|
||||
self._record_spoken(msg["id"])
|
||||
|
||||
def _record_spoken(self, rid: str) -> None:
|
||||
"""Keep a spoken answer once it is made and the app said how much of it was heard (whole
|
||||
sentences, the last one maybe cut off by a stop)."""
|
||||
rec = self.tts_recorded.get(rid)
|
||||
if rec is None or not rec["made"] or rec["played"] is None:
|
||||
return
|
||||
self.tts_recorded.pop(rid)
|
||||
heard = rec["pcm"][:rec["played"]]
|
||||
if heard and self.worker.recorder is not None:
|
||||
self.worker.recorder.spoke(rec["at"], rid, rec["text"], np.concatenate(heard), rec["rate"],
|
||||
rec["cancelled"])
|
||||
|
||||
def _tts_error(self, req: TtsRequest, message: str) -> None:
|
||||
self.error("tts_failed", message, id=req.id)
|
||||
|
||||
@@ -165,6 +165,7 @@ line (-0.6) would lose more real words than it saves.
|
||||
-> {"type": "tts", "id": "r1", "text": "...", "more": false} // more=true: more text follows for r1
|
||||
-> {"type": "tts_cancel", "id": "r1"}
|
||||
-> {"type": "tts_speed", "speed": 1.8} // 0.5-2.0, from the next sentence on
|
||||
-> {"type": "tts_played", "id": "r1", "played": 1, "cancelled": true} // how much the app played (Recordings)
|
||||
<- binary 0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
|
||||
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}
|
||||
```
|
||||
@@ -481,15 +482,20 @@ sets `JARVIS_RECORDINGS_DIR` (setting `voice.record`, off by default; the app pa
|
||||
|
||||
- each finished utterance (every final, also one with no text) as a 16 kHz mono WAV of the clip as it
|
||||
was cut for transcription, before loudness levelling: `<day>/<HHMMSS>-<ms>-said-<speaker>.wav`;
|
||||
- each spoken answer (every `tts` request that made sound) as a mono WAV at the session's TTS rate,
|
||||
as far as it was made (also when cancelled or failed): `<day>/<HHMMSS>-<ms>-jarvis.wav`, timed at
|
||||
its first sentence;
|
||||
- each spoken answer as a mono WAV at the session's TTS rate, as far as the app played it:
|
||||
`<day>/<HHMMSS>-<ms>-jarvis.wav`, timed at its first sentence. The app sends `tts_played` (`played`:
|
||||
how many of its sentences started playing, `cancelled`: whether it was stopped) when each request
|
||||
ends; the worker keeps the request's sentences until it is made and that report came, then writes
|
||||
the first `played` sentences (the last one maybe cut off by the stop; it may have been made in full)
|
||||
with `cancelled`. Nothing played (0), never reported (the session ends, an older app; at most 50
|
||||
are held) or ids it does not know (the earcon): no file;
|
||||
- one JSON line per file in `<day>/index.jsonl`: `{file, at, ms, kind: said|spoke, who, name,
|
||||
channel, text, id, cancelled}` (the transcript, or the text sent to `tts`).
|
||||
|
||||
`<day>` is the local day. Files are written on a thread of their own, so the voice path never waits,
|
||||
and a failed write is a debug line (no text) that changes nothing else. Day folders older than 7 days
|
||||
are removed when the worker starts. Nothing is recorded without the variable.
|
||||
are removed when the worker starts and with the first recording of each new day while it runs.
|
||||
Nothing is recorded without the variable.
|
||||
|
||||
### Errors
|
||||
|
||||
|
||||
Reference in New Issue
Block a user