diff --git a/app/src/admin.spec.md b/app/src/admin.spec.md index af1da01..79226ab 100644 --- a/app/src/admin.spec.md +++ b/app/src/admin.spec.md @@ -232,11 +232,14 @@ records and "Traces"), so it works while the services are stopped. Nothing it re `thought` (a trace: the question, the answer or none, how long it took and its steps). Jarvis's chat messages are known by the bot's user id (the first part of the stored token is that id in base64) and by the author of any chat answer to someone. -- `history_query` `{said, answered, thought, via, person, channel, find, since, until, limit}`: the +- `history_query` `{said, answered, thought, via, person, channel, find, since, until, before, from, limit}`: the three kinds (each on unless false), where (`voice`, `chat`, `task`; a task has traces only), a person (their records, answers to them and traces of their questions), a channel, text the record or the trace's question, answer or steps contains (records folded like the store: NFKC, lower - case), Unix seconds from (inclusive) and to (exclusive), and `limit` (1-2000, default 200). Answers + case), Unix seconds from (inclusive) and to (exclusive), the cursors `before` (only items older + than it) and `from` (it and newer), each an item's `{at, key}`, and `limit` (1-2000, default 200). + Items are ordered by time, then records before traces, then id, so paging by cursor never skips or + repeats an item, even at equal times. Answers `{items, more, people, channels}`: the newest `limit` items in time order, whether older ones were left out, the people who spoke (their latest name) and the channels with records (with the name given a chat channel in the settings). Without the file: no items and `missing: true`. @@ -684,7 +687,7 @@ back as Korean messages for the screen. The Tauri shell passes the screens' `inv | `service_start` / `service_stop` / `service_restart` / `dashboard` | the controller and its snapshot; start and restart run in the background, stop also cancels a start | | `discord_channel_info` `{id}` / `channel_sync_status` | a channel's Discord name and topic / each synced chat channel's last result (above) | | `notes_list` / `note_update` `{id, text}` / `note_delete` `{id}` | everyone's personal notes (above) / change one's text / delete one | -| `history_query` `{said, answered, thought, via, person, channel, find, since, until, limit}` | the 대화 screen's records and answer traces (above, "Conversations") | +| `history_query` `{said, answered, thought, via, person, channel, find, since, until, before, from, limit}` | the 대화 screen's records and answer traces (above, "Conversations") | | `activity_list` `{limit}` | what Jarvis did (above, "Activity"), newest first | | `schedules_list` / `schedule_delete` `{id}` | every person's schedules (above), soonest first, with IDs as strings / delete one | | `collect_results` `{target, limit}` | `{targets, items}`: every target folder with results, and the collected items newest first (`target` null: every target; `limit` 1-500, default 100) | @@ -736,9 +739,12 @@ Plain HTML, CSS and an ES module, no build step, dark theme after `DESIGN.md`. A notes it was given, the tools offered, the plan, the model's thinking, each tool call (name, arguments, success, failure or skipped and why, its result) and the answer. Filters: the three kinds as chips, where (음성, 채팅, 정기 작업), person, channel, one day and text; "실시간" (on by - default) refreshes every 3 seconds and redraws only when something changed, keeping the cards the - user opened open and the scroll where it was unless it was at the end; "이전 기록 더 보기" adds 200 - older items; "생각 모두 펼치기/접기"; a count of each kind shown. + default) re-reads every item from the oldest one shown on (`from`, up to 2,000; older pages beyond + that stay as loaded) every 3 seconds and redraws only when an item changed, an edited message + included, keeping the cards the user opened open and the scroll where it was unless it was at the + end; "이전 기록 더 보기" adds the 200 items before the oldest shown (`before`), with no end to how far + back it goes; a filter change drops every answer to a request made before it, and a live answer + older than the newest one asked is dropped; "생각 모두 펼치기/접기"; a count of each kind shown. - **한 일**: the activity (time, kind, target, who asked, outcome and why it failed), newest 200, with a refresh button. - **업데이트**: the version here, the state, "다시 확인", "받기" (with its percentage) and "설치하고 diff --git a/app/src/admin_api.rs b/app/src/admin_api.rs index a0760ef..3c90bc5 100644 --- a/app/src/admin_api.rs +++ b/app/src/admin_api.rs @@ -405,6 +405,8 @@ impl AdminApi { said: flag("said"), answered: flag("answered"), thought: flag("thought"), via: text("via"), person: text("person"), channel: text("channel"), find: text("find"), since: args.get("since").and_then(Value::as_f64), until: args.get("until").and_then(Value::as_f64), + before: args.get("before").and_then(crate::history_view::Cursor::of), + from: args.get("from").and_then(crate::history_view::Cursor::of), limit: args.get("limit").and_then(Value::as_u64).map_or(crate::history_view::DEFAULT_LIMIT, |n| n as usize), }; let bot = self.secrets()?.get(DISCORD_TOKEN).ok().flatten().and_then(|t| crate::history_view::bot_id_of_token(&t)); diff --git a/app/src/history_view.rs b/app/src/history_view.rs index 444aff9..1c37bdb 100644 --- a/app/src/history_view.rs +++ b/app/src/history_view.rs @@ -32,13 +32,61 @@ pub struct Query { /// Unix seconds, inclusive / exclusive. pub since: Option, pub until: Option, + /// Only items older than this one (paging back) / this one and newer (the window shown). + pub before: Option, + pub from: Option, pub limit: usize, } +/// An item's place in the order: time, then records before traces, then id. Items are ordered by +/// it, so paging by it never skips or repeats one, even when two share a time. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct Cursor { + pub at: f64, + /// 0: a record (`h`), 1: a trace (`t`). + pub src: u8, + pub id: i64, +} + +impl Cursor { + /// From `{at, key}`, as the screen sends an item back. + pub fn of(v: &Value) -> Option { + let at = v.get("at")?.as_f64()?; + let key = v.get("key")?.as_str()?; + let src = match key.get(..1)? { "h" => 0, "t" => 1, _ => return None }; + Some(Self { at, src, id: key[1..].parse().ok()? }) + } + + /// SQL for "a row of table `src` comes before this cursor", over columns `{p}ts` and `{p}id`. + fn before_sql(&self, src: u8, p: &str) -> (String, Vec) { + use std::cmp::Ordering::*; + match src.cmp(&self.src) { + Less => (format!("{p}ts <= ?"), vec![Sql::Real(self.at)]), + Greater => (format!("{p}ts < ?"), vec![Sql::Real(self.at)]), + Equal => (format!("({p}ts < ? OR ({p}ts = ? AND {p}id < ?))"), + vec![Sql::Real(self.at), Sql::Real(self.at), Sql::Integer(self.id)]), + } + } +} + +/// The cursor conditions of `q` for table `src`, appended to `sql` and `args`. +fn cursor_sql(q: &Query, src: u8, p: &str, sql: &mut String, args: &mut Vec) { + if let Some(c) = &q.before { + let (cond, a) = c.before_sql(src, p); + *sql += &format!(" AND {cond}"); + args.extend(a); + } + if let Some(c) = &q.from { + let (cond, a) = c.before_sql(src, p); + *sql += &format!(" AND NOT {cond}"); + args.extend(a); + } +} + impl Default for Query { fn default() -> Self { Self { said: true, answered: true, thought: true, via: None, person: None, channel: None, find: None, - since: None, until: None, limit: DEFAULT_LIMIT } + since: None, until: None, before: None, from: None, limit: DEFAULT_LIMIT } } } @@ -98,7 +146,7 @@ pub fn query(path: &Path, bot: Option<&str>, q: &Query) -> Result }; let name_of = |id: &str| names.get(id).cloned().unwrap_or_default(); - let mut items: Vec<(f64, Value)> = Vec::new(); + let mut items: Vec<((f64, u8, i64), Value)> = Vec::new(); let mut more = false; if q.said || q.answered { let mut sql = format!("SELECT h.id, h.ts, h.kind, h.channel_id, h.author_id, h.author_name, h.to_id, h.text, \ @@ -129,13 +177,14 @@ pub fn query(path: &Path, bot: Option<&str>, q: &Query) -> Result } if let Some(t) = q.since { sql += " AND h.ts >= ?"; args.push(Sql::Real(t)); } if let Some(t) = q.until { sql += " AND h.ts < ?"; args.push(Sql::Real(t)); } + cursor_sql(q, 0, "h.", &mut sql, &mut args); sql += &format!(" ORDER BY h.ts DESC, h.id DESC LIMIT {}", limit + 1); let mut st = c.prepare(&sql).map_err(err)?; let rows = st.query_map(params_from_iter(args), |r| { let (id, ts, kind): (i64, f64, String) = (r.get(0)?, r.get(1)?, r.get(2)?); let (author, author_name, to): (String, String, Option) = (r.get(4)?, r.get(5)?, r.get(6)?); let answer: bool = r.get(8)?; - Ok((ts, json!({ + Ok(((ts, 0, id), json!({ "key": format!("h{id}"), "type": if answer { "answered" } else { "said" }, "at": ts, "via": if kind == "chat" { "chat" } else { "voice" }, "channel": r.get::<_, String>(3)?, "who": {"id": author, "name": author_name}, @@ -159,13 +208,14 @@ pub fn query(path: &Path, bot: Option<&str>, q: &Query) -> Result } if let Some(t) = q.since { sql += " AND ts >= ?"; args.push(Sql::Real(t)); } if let Some(t) = q.until { sql += " AND ts < ?"; args.push(Sql::Real(t)); } + cursor_sql(q, 1, "", &mut sql, &mut args); let sql = format!("{sql} ORDER BY ts DESC, id DESC LIMIT {}", limit + 1); let mut st = c.prepare(&sql).map_err(err)?; let rows = st.query_map(params_from_iter(args), |r| { let (id, ts): (i64, f64) = (r.get(0)?, r.get(1)?); let user: String = r.get(4)?; let steps: String = r.get(8)?; - Ok((ts, json!({ + Ok(((ts, 1, id), json!({ "key": format!("t{id}"), "type": "thought", "at": ts, "via": r.get::<_, String>(2)?, "channel": r.get::<_, Option>(3)?, "who": {"id": user.clone(), "name": name_of(&user)}, @@ -177,7 +227,7 @@ pub fn query(path: &Path, bot: Option<&str>, q: &Query) -> Result more |= got.len() > limit; items.extend(got.into_iter().take(limit)); } - items.sort_by(|a, b| b.0.total_cmp(&a.0)); + items.sort_by(|a, b| b.0.0.total_cmp(&a.0.0).then(b.0.1.cmp(&a.0.1)).then(b.0.2.cmp(&a.0.2))); more |= items.len() > limit; items.truncate(limit); items.reverse(); @@ -305,6 +355,35 @@ mod tests { assert_eq!(v["more"], true); } + /// Every item, walking back page by page with the cursor of the oldest one shown. + #[test] + fn pages_walk_back_through_everything_past_any_limit_even_at_equal_times() { + let (path, c) = db("pages"); + for i in 0..7 { + // Records and traces at the same moments, so a page edge falls inside equal times. + rec(&c, (i / 2) as f64, "voice", "vc", "1", "Alice", None, &format!("말 {i}")); + trace(&c, (i / 2) as f64, "voice", Some("vc"), "1", &format!("생각 {i}"), "", "[]"); + } + let all = query(&path, None, &Query { limit: MAX_LIMIT, ..Query::default() }).unwrap()["items"].clone(); + let all: Vec = all.as_array().unwrap().clone(); + assert_eq!(all.len(), 14); + let cursor = |i: &Value| Cursor::of(&json!({"at": i["at"], "key": i["key"]})).unwrap(); + let mut got: Vec = Vec::new(); + let mut before = None; + loop { + let v = query(&path, None, &Query { limit: 3, before, ..Query::default() }).unwrap(); + let page = v["items"].as_array().unwrap().clone(); + before = page.first().map(cursor); + got.splice(0..0, page); + if v["more"] == false { break; } + } + assert_eq!(got, all); + // From a cursor on: that item and everything newer, whatever the limit was before. + let from = query(&path, None, &Query { limit: MAX_LIMIT, from: Some(cursor(&all[5])), ..Query::default() }).unwrap(); + assert_eq!(from["items"].as_array().unwrap()[..], all[5..]); + assert!(Cursor::of(&json!({"at": 1.0, "key": "x1"})).is_none()); + } + #[test] fn the_bot_id_comes_from_its_token() { assert_eq!(bot_id_of_token("MTIzNDU2Nzg5MDEyMzQ1Njc4.GxYzAb.abcdef").as_deref(), Some("123456789012345678")); diff --git a/app/ui/app.js b/app/ui/app.js index 37fde9f..7c9092e 100644 --- a/app/ui/app.js +++ b/app/ui/app.js @@ -446,10 +446,16 @@ function conversationScreen() { const find = h("input", { type: "text", placeholder: "찾기 (말, 답, 생각)", style: "max-width:220px" }); const live = h("input", { type: "checkbox", class: "switch", checked: true }); const open = new Set(); // traces the user opened stay open across refreshes - let expandAll = false, limit = 200, sig = null; + // The window shown: every item from the cursor `from` on, re-read live (at most PAGE_MAX), plus + // `older` pages that no longer fit it. Each filter change starts a new epoch; answers from an + // earlier epoch, and live answers older than the newest one asked, are dropped. + const PAGE = 200, PAGE_MAX = 2000; + const cursorOf = (i) => ({ at: i.at, key: i.key }); + let expandAll = false, sig = null, shown = [], older = [], from = null, hasOlder = false, missing = false; + let epoch = 0, liveSeq = 0, first = true; const box = h("div", { class: "talk" }); const summary = h("div", { class: "small muted" }); - const more = h("button", { class: "btn", hidden: true, onclick: () => { limit += 200; load(true); } }, "이전 기록 더 보기"); + const more = h("button", { class: "btn", hidden: true, onclick: () => loadOlder() }, "이전 기록 더 보기"); const fill = (sel, list, label) => { const key = JSON.stringify(list); @@ -486,37 +492,78 @@ function conversationScreen() { h("div", { class: "text" }, i.text)); } - async function load(keepTop = false) { + function filters() { + const since = day.value ? new Date(day.value + "T00:00:00").getTime() / 1000 : null; + return { said: said.checked, answered: answered.checked, thought: thought.checked, + via: via.value || null, person: person.value || null, channel: channel.value || null, find: find.value || null, + since, until: since == null ? null : since + 86400 }; + } + + function render(keepTop = false) { + const items = older.concat(shown); + // The whole items, so an edited message or a changed answer is redrawn too. + const s = JSON.stringify([items, expandAll, hasOlder, missing]); + if (s === sig) return; + sig = s; + const stick = box.scrollTop + box.clientHeight >= box.scrollHeight - 30, fromBottom = box.scrollHeight - box.scrollTop; + const kids = []; + let lastDay = null; + for (const i of items) { + const d = dayOf(i.at); + if (d !== lastDay) { kids.push(h("div", { class: "day" }, d)); lastDay = d; } + kids.push(item(i)); + } + box.replaceChildren(...(kids.length ? kids : [h("div", { class: "muted" }, + missing ? "아직 대화 기록이 없습니다. 서비스를 켜고 자비스와 이야기하면 여기에 나옵니다." : "조건에 맞는 기록이 없습니다.")])); + more.hidden = !hasOlder; + const n = (t) => items.filter((i) => i.type === t).length; + summary.textContent = `사람 말 ${n("said")} · 자비스 말 ${n("answered")} · 생각 ${n("thought")}` + (hasOlder ? " (이전 기록 더 있음)" : ""); + if (keepTop) box.scrollTop = box.scrollHeight - fromBottom; + else if (stick || first) box.scrollTop = box.scrollHeight; + first = false; + } + + function lists(r) { + load.channels = r.channels; + fill(person, r.people, (p) => `${p.name || p.id}`); + fill(channel, r.channels, channelLabel); + } + + /** The window from `from` on (the newest page at first), re-read on every live tick. */ + async function load() { + const ep = epoch, my = ++liveSeq; try { - const since = day.value ? new Date(day.value + "T00:00:00").getTime() / 1000 : null; - const r = await api("history_query", { said: said.checked, answered: answered.checked, thought: thought.checked, - via: via.value || null, person: person.value || null, channel: channel.value || null, find: find.value || null, - since, until: since == null ? null : since + 86400, limit }); - load.channels = r.channels; - fill(person, r.people, (p) => `${p.name || p.id}`); - fill(channel, r.channels, channelLabel); - const s = JSON.stringify([r.items.map((i) => i.key + ":" + (i.text || i.answer || "").length), expandAll, r.more]); - if (s === sig) return; - sig = s; - const stick = box.scrollTop + box.clientHeight >= box.scrollHeight - 30, fromBottom = box.scrollHeight - box.scrollTop; - const kids = []; - let lastDay = null; - for (const i of r.items) { - const d = dayOf(i.at); - if (d !== lastDay) { kids.push(h("div", { class: "day" }, d)); lastDay = d; } - kids.push(item(i)); - } - box.replaceChildren(...(kids.length ? kids : [h("div", { class: "muted" }, - r.missing ? "아직 대화 기록이 없습니다. 서비스를 켜고 자비스와 이야기하면 여기에 나옵니다." : "조건에 맞는 기록이 없습니다.")])); - more.hidden = !r.more; - const n = (t) => r.items.filter((i) => i.type === t).length; - summary.textContent = `사람 말 ${n("said")} · 자비스 말 ${n("answered")} · 생각 ${n("thought")}` + (r.more ? ` (최근 ${limit}개까지)` : ""); - if (keepTop) box.scrollTop = box.scrollHeight - fromBottom; - else if (stick || !load.done) box.scrollTop = box.scrollHeight; - load.done = true; + const r = await api("history_query", { ...filters(), ...(from ? { from, limit: PAGE_MAX } : { limit: PAGE }) }); + if (ep !== epoch || my !== liveSeq) return; + lists(r); + missing = !!r.missing; + if (!from) hasOlder = r.more; + else if (r.more) { older = []; hasOlder = true; } // more new ones than a window holds: page back from them + shown = r.items; + if (shown.length) from = cursorOf(shown[0]); + render(); } catch (e) { fail(e); } } - const reload = () => { limit = 200; sig = null; load.done = false; load(); }; + + /** The page before the oldest item shown; it joins the live window while that stays under PAGE_MAX. */ + async function loadOlder() { + const oldest = older[0] || shown[0]; + if (!oldest) return; + const ep = epoch; + try { + const r = await api("history_query", { ...filters(), before: cursorOf(oldest), limit: PAGE }); + if (ep !== epoch) return; + ++liveSeq; // a live answer asked with the old window would drop this page + hasOlder = r.more; + if (!older.length && shown.length + r.items.length <= PAGE_MAX) { + shown = r.items.concat(shown); + if (shown.length) from = cursorOf(shown[0]); + } else older = r.items.concat(older); + render(true); + } catch (e) { fail(e); } + } + + const reload = () => { epoch++; sig = null; shown = []; older = []; from = null; hasOlder = false; first = true; load(); }; for (const el of [said, answered, thought, via, person, channel, day]) el.addEventListener("change", reload); find.addEventListener("input", () => { clearTimeout(find.t); find.t = setTimeout(reload, 300); }); const view = h("div", {}, h("h1", {}, "대화"),