feat(app): background collection frame for sites, search words and RSS feeds
Targets (none by default), interval and keep days on a new 백그라운드 수집 screen with the results; the worker collects on its own thread through the web search's address check and writes only to the data folder's collect/ (logs carry target names and counts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -38,9 +38,9 @@ that is not absolute, or is a file, is refused with a message.
|
||||
`config/settings.json` keeps the format of `discord_core.spec.md` and `conversation.spec.md`.
|
||||
|
||||
- `fields()` describes every key the screens edit: its dotted path (`call.mode`), screen
|
||||
(`bot`, `channels`, `chat`, `memory`, `model`, `features`), kind (bool, integer with a range, number with a
|
||||
(`bot`, `channels`, `chat`, `memory`, `model`, `features`, `collect`), kind (bool, integer with a range, number with a
|
||||
range, text, text list, choice with its values, Discord ID, Discord ID list, ID to number map,
|
||||
chat channel list, MCP server list, free object), its default and a Korean label. The screens are built from this
|
||||
chat channel list, MCP server list, collection target list, free object), its default and a Korean label. The screens are built from this
|
||||
list, so a key is added in one place.
|
||||
- `view(dir)` returns every field's current value: the file's value, or the default when the file
|
||||
does not set it. A file that does not parse is reported, not overwritten.
|
||||
@@ -59,10 +59,11 @@ that is not absolute, or is a file, is refused with a message.
|
||||
manages the MCP servers it connects.
|
||||
|
||||
- `features` (new section): `weather` (`getWeather`), `webSearch` (`webSearch`, `fetchWebPage`) and
|
||||
`meals` (`logMeal`, `fetchMeals`, `deleteMeal`), each on by default. The tools of a feature that
|
||||
`meals` (`logMeal`, `fetchMeals`, `deleteMeal`), each on by default; `collect` (background
|
||||
collection, below; no brain tool), also on by default. The tools of a feature that
|
||||
is off go into the brain's `disabled_tools`: they are neither offered nor run
|
||||
(`voice_worker.spec.md`, "Tools"). The other features of the 기능 screen (browser control, PC
|
||||
control, screen viewing, background collection) come with their milestones (M6, M7).
|
||||
control, screen viewing) come with their milestones (M6, M7).
|
||||
- `mcpServers` (new list): each `{name, command, args, env, enabled}`. `name` is 1-32 letters,
|
||||
digits, `-` or `_` without `__` (the brain names a server's tools `<name>__<tool>`), not listed
|
||||
twice; `command` is not empty; `args` a list of strings; `env` the names of the environment
|
||||
@@ -79,6 +80,26 @@ manages the MCP servers it connects.
|
||||
also without debug logging. MCP servers are programs the PC runs; the screen says to add only
|
||||
trusted ones.
|
||||
|
||||
## Background collection (`app/src/collect.rs`)
|
||||
|
||||
`docs/v2/desktop-app.md`, "백그라운드 수집", and `docs/v2/pc-features.md`. The frame for collecting
|
||||
sites, search words and RSS feeds on a schedule; there are **no targets by default**.
|
||||
|
||||
- `collect` (new section): `keepDays` (1-365, default 30) and `targets` (default empty), each
|
||||
`{name, kind, value, everyMinutes, enabled}`. `name` is 1-32 letters, digits, `-` or `_` (any
|
||||
script; it names the target's folder, so not a Windows device name like `con` or `nul`), not
|
||||
listed twice; `kind` is `site`, `search` or `rss`; `value` is an http(s) address for `site` and
|
||||
`rss` (at most 2,000 characters) and the words for `search` (1-200 characters);
|
||||
`everyMinutes` 15-10080 (default 60); `enabled` on by default.
|
||||
- On start, with `features.collect` on and at least one enabled target, the app writes
|
||||
`config/collect.json` (the folder `collect/`, `keepDays` and the enabled targets) and passes it to
|
||||
the worker as `JARVIS_COLLECT_PATH`; otherwise it removes that file and passes nothing. The worker
|
||||
collects while the services run (`voice_worker.spec.md`, "Background collection"); results stay
|
||||
in `collect/<name>/<YYYY-MM-DD>.jsonl` in the data folder only.
|
||||
- `results(data, target, limit)` reads them newest first (the newest day file first, its last line
|
||||
first), each item with its `target`; a name that is not a valid target name is refused, so no
|
||||
other folder is read. Lines that do not parse are skipped.
|
||||
|
||||
## Secrets (`app/src/secrets.rs`)
|
||||
|
||||
`config/secrets.dat` holds a JSON object of named secrets (`discordToken`, the MCP servers'
|
||||
@@ -284,6 +305,7 @@ back as Korean messages for the screen. The Tauri shell passes the screens' `inv
|
||||
| `settings_fields` / `settings_view` / `settings_apply` `{changes}` | the settings editor above; `restartNeeded` when the services run; `sync` with the chat channels sent to Discord (below) |
|
||||
| `service_start` / `service_stop` / `service_restart` / `dashboard` | the controller and its snapshot; start and restart run in the background, stop also cancels a start |
|
||||
| `discord_channel_info` `{id}` / `channel_sync_status` | a channel's Discord name and topic / each synced chat channel's last result (above) |
|
||||
| `collect_results` `{target, limit}` | collected items newest first (`target` null: every target; `limit` 1-500, default 100) |
|
||||
| `mcp_env_status` / `mcp_env_set` `{server, name, value}` | which MCP environment values are stored (`{server: {NAME: bool}}`, never the values) / store one (an empty value removes it); only for a saved server and name |
|
||||
| `install_status` / `install_start` / `install_cancel` | the install steps and their state, bytes and last line / run what is not done in the background / stop it |
|
||||
| `logs_query` `{tab, level, find, after, limit}` / `logs_error_report` / `logs_export` `{from, to}` | the log hub; the zip goes to `logs/` and its path comes back |
|
||||
@@ -309,13 +331,16 @@ Plain HTML, CSS and an ES module, no build step, dark theme after `DESIGN.md`. A
|
||||
Strata's last line, and stop in the header; the header with start, restart and stop is rebuilt
|
||||
only when the state changes, so a click is never lost to the refresh. Every screen change starts
|
||||
a new generation: an answer that arrives after the user moved to another screen is dropped.
|
||||
- **Settings screens** (봇, 채널·호출, 채팅 채널, 대화 기억, 모델, 기능): built from `settings_fields`; only
|
||||
- **Settings screens** (봇, 채널·호출, 채팅 채널, 대화 기억, 모델, 기능, 백그라운드 수집): built from `settings_fields`; only
|
||||
changed fields are sent; the server and voice channel are pickers (choosing another server clears
|
||||
the voice channel and lists only that server's channels); the chat channels are a table
|
||||
with their switches; the bot screen also changes the token; the 기능 screen has the feature
|
||||
switches and one card per MCP server (name, command, arguments and environment variable names
|
||||
one per line, on/off, delete), and for a saved server a password box per environment variable
|
||||
that stores its value on its own.
|
||||
that stores its value on its own. The 백그라운드 수집 screen has the targets (a table: name, kind,
|
||||
address or words, interval in minutes, on/off, delete), the keep days, and the results (the
|
||||
newest 100, filtered by target, each with its time, title, link and the start of its text shown
|
||||
as plain text).
|
||||
- **Logs**: tab, level and text filters, pause, live follow (lines after the last seen), "최근 오류
|
||||
복사" (to the clipboard; over 2,000 characters it suggests the zip), and the zip export by days.
|
||||
|
||||
|
||||
@@ -281,6 +281,11 @@ impl AdminApi {
|
||||
Ok(json!({"set": !value.is_empty(),
|
||||
"restartNeeded": self.controller()?.state() == crate::control::State::Running}))
|
||||
}
|
||||
"collect_results" => {
|
||||
let target = args.get("target").and_then(Value::as_str).filter(|t| !t.is_empty());
|
||||
let limit = args.get("limit").and_then(Value::as_u64).unwrap_or(100).clamp(1, 500) as usize;
|
||||
Ok(json!(crate::collect::results(self.dir()?.path(), target, limit)?))
|
||||
}
|
||||
"channel_sync_status" => Ok(Value::Object(self.syncer()?.status().into_iter()
|
||||
.map(|(id, s)| (id.to_string(), json!(s))).collect())),
|
||||
"discord_channel_info" => {
|
||||
@@ -392,6 +397,22 @@ mod tests {
|
||||
assert!(a.call("nope", json!({})).await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn collection_targets_are_saved_and_their_results_read_from_the_data_folder() {
|
||||
let (a, base) = api("api-collect");
|
||||
let data = base.join("d");
|
||||
a.call("folder_choose", json!({"path": data.to_string_lossy()})).await.unwrap();
|
||||
let targets = json!([{"name": "뉴스", "kind": "rss", "value": "https://feed.example/rss", "everyMinutes": 30}]);
|
||||
a.call("settings_apply", json!({"changes": {"collect.targets": targets, "collect.keepDays": 7}})).await.unwrap();
|
||||
assert_eq!(a.call("settings_view", json!({})).await.unwrap()["collect.targets"], targets);
|
||||
let day = data.join("collect").join("뉴스");
|
||||
std::fs::create_dir_all(&day).unwrap();
|
||||
std::fs::write(day.join("2026-11-13.jsonl"), "{\"at\": \"2026-11-13T09:00:00+09:00\", \"title\": \"t\"}\n").unwrap();
|
||||
let r = a.call("collect_results", json!({"target": null})).await.unwrap();
|
||||
assert_eq!(r, json!([{"at": "2026-11-13T09:00:00+09:00", "title": "t", "target": "뉴스"}]));
|
||||
assert!(a.call("collect_results", json!({"target": "../config"})).await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mcp_values_are_stored_encrypted_shown_only_as_set_masked_in_logs_and_go_with_their_server() {
|
||||
let (a, base) = api("api-mcp");
|
||||
|
||||
131
app/src/collect.rs
Normal file
131
app/src/collect.rs
Normal file
@@ -0,0 +1,131 @@
|
||||
//! Background collection (`app/src/admin.spec.md`, "Background collection"): the worker's
|
||||
//! collection file, and the collected items for the 백그라운드 수집 screen.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use serde_json::{json, Value};
|
||||
|
||||
use crate::settings::{valid_collect_name, Settings};
|
||||
|
||||
/// The worker's environment variable with the collection file (`voice_worker/collect.py`).
|
||||
pub const COLLECT_ENV_VAR: &str = "JARVIS_COLLECT_PATH";
|
||||
|
||||
/// Write `config/collect.json` with the enabled targets and return it; with the feature off or no
|
||||
/// enabled target, remove it and return None (the worker then collects nothing).
|
||||
pub fn write_config(data: &Path, settings: &Settings) -> std::io::Result<Option<PathBuf>> {
|
||||
let file = data.join("config").join("collect.json");
|
||||
let targets: Vec<Value> = settings.collect.targets.iter().filter(|t| t.enabled)
|
||||
.map(|t| json!({"name": t.name, "kind": t.kind, "value": t.value, "everyMinutes": t.every_minutes}))
|
||||
.collect();
|
||||
if !settings.features.collect || targets.is_empty() {
|
||||
match std::fs::remove_file(&file) {
|
||||
Err(e) if e.kind() != std::io::ErrorKind::NotFound => return Err(e),
|
||||
_ => return Ok(None),
|
||||
}
|
||||
}
|
||||
let cfg = json!({"dir": data.join("collect").to_string_lossy(), "keepDays": settings.collect.keep_days,
|
||||
"targets": targets});
|
||||
let mut text = serde_json::to_string_pretty(&cfg).unwrap();
|
||||
text.push('\n');
|
||||
crate::datadir::write_atomic(&file, text.as_bytes())?;
|
||||
Ok(Some(file))
|
||||
}
|
||||
|
||||
/// The day files of one target folder, newest first.
|
||||
fn day_files(folder: &Path) -> Vec<PathBuf> {
|
||||
let mut files: Vec<PathBuf> = std::fs::read_dir(folder).into_iter().flatten().flatten().map(|e| e.path())
|
||||
.filter(|p| p.file_name().and_then(|n| n.to_str()).is_some_and(|n| n.len() == 16 && n.ends_with(".jsonl")))
|
||||
.collect();
|
||||
files.sort();
|
||||
files.reverse();
|
||||
files
|
||||
}
|
||||
|
||||
/// Collected items newest first, each with its `target`; `target` None reads every target folder.
|
||||
pub fn results(data: &Path, target: Option<&str>, limit: usize) -> Result<Vec<Value>, String> {
|
||||
let root = data.join("collect");
|
||||
let names: Vec<String> = match target {
|
||||
Some(t) if !valid_collect_name(t) => return Err(format!("not a collection target name: {t}")),
|
||||
Some(t) => vec![t.to_owned()],
|
||||
None => std::fs::read_dir(&root).into_iter().flatten().flatten()
|
||||
.filter(|e| e.path().is_dir())
|
||||
.filter_map(|e| e.file_name().to_str().map(str::to_owned))
|
||||
.filter(|n| valid_collect_name(n))
|
||||
.collect(),
|
||||
};
|
||||
let mut out: Vec<Value> = Vec::new();
|
||||
for name in names {
|
||||
let mut n = 0;
|
||||
'files: for f in day_files(&root.join(&name)) {
|
||||
let text = std::fs::read_to_string(&f).unwrap_or_default();
|
||||
for line in text.lines().rev() {
|
||||
if let Ok(Value::Object(mut item)) = serde_json::from_str::<Value>(line) {
|
||||
item.insert("target".into(), Value::String(name.clone()));
|
||||
out.push(Value::Object(item));
|
||||
n += 1;
|
||||
if n == limit {
|
||||
break 'files;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// `at` is the local time with its offset in one format, so the text order is the time order.
|
||||
out.sort_by(|a, b| b["at"].as_str().unwrap_or("").cmp(a["at"].as_str().unwrap_or("")));
|
||||
out.truncate(limit);
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::datadir::temp_dir;
|
||||
use crate::settings::parse_settings;
|
||||
|
||||
fn settings(v: Value) -> Settings {
|
||||
parse_settings(&v).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_worker_gets_only_enabled_targets_and_nothing_without_them() {
|
||||
let data = temp_dir("collect-cfg");
|
||||
let s = settings(json!({"collect": {"keepDays": 3, "targets": [
|
||||
{"name": "a", "kind": "rss", "value": "https://a.example/rss", "everyMinutes": 30},
|
||||
{"name": "b", "kind": "site", "value": "https://b.example", "enabled": false}]}}));
|
||||
let file = write_config(&data, &s).unwrap().unwrap();
|
||||
let cfg: Value = serde_json::from_str(&std::fs::read_to_string(&file).unwrap()).unwrap();
|
||||
assert_eq!(cfg, json!({"dir": data.join("collect").to_string_lossy(), "keepDays": 3,
|
||||
"targets": [{"name": "a", "kind": "rss", "value": "https://a.example/rss", "everyMinutes": 30}]}));
|
||||
|
||||
let mut off = s.clone();
|
||||
off.features.collect = false;
|
||||
assert_eq!(write_config(&data, &off).unwrap(), None);
|
||||
assert!(!file.exists());
|
||||
write_config(&data, &s).unwrap();
|
||||
assert_eq!(write_config(&data, &settings(json!({}))).unwrap(), None);
|
||||
assert!(!file.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn results_come_newest_first_with_their_target_and_only_from_target_folders() {
|
||||
let data = temp_dir("collect-res");
|
||||
let put = |name: &str, day: &str, lines: &[&str]| {
|
||||
let d = data.join("collect").join(name);
|
||||
std::fs::create_dir_all(&d).unwrap();
|
||||
std::fs::write(d.join(format!("{day}.jsonl")), lines.join("\n") + "\n").unwrap();
|
||||
};
|
||||
put("뉴스", "2026-11-12", &[r#"{"at": "2026-11-12T09:00:00+09:00", "title": "old"}"#]);
|
||||
put("뉴스", "2026-11-13", &[r#"{"at": "2026-11-13T09:00:00+09:00", "title": "n1"}"#, "not json",
|
||||
r#"{"at": "2026-11-13T10:00:00+09:00", "title": "n2"}"#]);
|
||||
put("site", "2026-11-13", &[r#"{"at": "2026-11-13T09:30:00+09:00", "title": "s1"}"#]);
|
||||
std::fs::write(data.join("collect").join("뉴스").join("state.json"), r#"{"at": "9999"}"#).unwrap();
|
||||
let titles = |v: Vec<Value>| v.iter().map(|i| format!("{}:{}", i["target"].as_str().unwrap(),
|
||||
i["title"].as_str().unwrap())).collect::<Vec<_>>();
|
||||
assert_eq!(titles(results(&data, None, 100).unwrap()), ["뉴스:n2", "site:s1", "뉴스:n1", "뉴스:old"]);
|
||||
assert_eq!(titles(results(&data, Some("뉴스"), 2).unwrap()), ["뉴스:n2", "뉴스:n1"]);
|
||||
assert_eq!(titles(results(&data, None, 2).unwrap()), ["뉴스:n2", "site:s1"]);
|
||||
assert!(results(&data, Some("missing"), 10).unwrap().is_empty());
|
||||
assert!(results(&data, Some(".."), 10).is_err());
|
||||
assert!(results(&data, Some("../config"), 10).is_err());
|
||||
}
|
||||
}
|
||||
@@ -100,9 +100,9 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn features_that_are_off_turn_their_tools_off() {
|
||||
let all_on = FeatureSettings { weather: true, web_search: true, meals: true };
|
||||
let all_on = FeatureSettings { weather: true, web_search: true, meals: true, collect: true };
|
||||
assert!(disabled_tools(&all_on).is_empty());
|
||||
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false });
|
||||
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false, collect: false });
|
||||
assert_eq!(off, vec!["getWeather", "logMeal", "fetchMeals", "deleteMeal"]);
|
||||
}
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ pub mod admin_api;
|
||||
pub mod calls;
|
||||
pub mod channel_sync;
|
||||
pub mod chat;
|
||||
pub mod collect;
|
||||
pub mod control;
|
||||
pub mod conversation;
|
||||
pub mod dashboard;
|
||||
|
||||
@@ -70,6 +70,12 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
|
||||
}
|
||||
wopts.env.insert("JARVIS_CONFIG_PATH".into(), brain_config.to_string_lossy().into_owned());
|
||||
wopts.env.insert(crate::features::MCP_ENV_VAR.into(), mcp_env.to_string());
|
||||
// Background collection: only with the feature on and enabled targets (admin.spec.md).
|
||||
match crate::collect::write_config(data_dir, &settings) {
|
||||
Ok(Some(f)) => { wopts.env.insert(crate::collect::COLLECT_ENV_VAR.into(), f.to_string_lossy().into_owned()); }
|
||||
Ok(None) => {}
|
||||
Err(e) => return Err(after_llm_stop(&llm, format!("could not write the collection settings: {e}")).await.into()),
|
||||
}
|
||||
let worker_process = match WorkerProcess::new(wopts) {
|
||||
Ok(p) => p,
|
||||
Err(e) => return Err(after_llm_stop(&llm, e.to_string()).await.into()),
|
||||
|
||||
@@ -91,6 +91,25 @@ pub struct FeatureSettings {
|
||||
pub weather: bool,
|
||||
pub web_search: bool,
|
||||
pub meals: bool,
|
||||
/// Background collection (admin.spec.md, "Background collection").
|
||||
pub collect: bool,
|
||||
}
|
||||
|
||||
/// Background collection (admin.spec.md, "Background collection"); no targets by default.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct CollectSettings {
|
||||
pub keep_days: u32,
|
||||
pub targets: Vec<CollectTarget>,
|
||||
}
|
||||
|
||||
/// One thing to collect: `kind` is `site`, `search` or `rss`; `value` its address or words.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct CollectTarget {
|
||||
pub name: String,
|
||||
pub kind: String,
|
||||
pub value: String,
|
||||
pub every_minutes: u32,
|
||||
pub enabled: bool,
|
||||
}
|
||||
|
||||
/// An MCP server the brain starts (stdio). The values of `env` are in the secrets, never here.
|
||||
@@ -121,6 +140,7 @@ pub struct Settings {
|
||||
pub llm: LlmSettings,
|
||||
pub features: FeatureSettings,
|
||||
pub mcp_servers: Vec<McpServerSettings>,
|
||||
pub collect: CollectSettings,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
@@ -164,8 +184,9 @@ impl Default for Settings {
|
||||
response: PolicySettings::default(),
|
||||
memory: Map::new(),
|
||||
llm: LlmSettings { url: None, port: 8080, context: 32768 },
|
||||
features: FeatureSettings { weather: true, web_search: true, meals: true },
|
||||
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true },
|
||||
mcp_servers: vec![],
|
||||
collect: CollectSettings { keep_days: 30, targets: vec![] },
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -483,12 +504,76 @@ pub fn parse_settings(raw: &Value) -> Result<Settings, SettingsError> {
|
||||
weather: get_bool(f, "weather", "features", d.features.weather)?,
|
||||
web_search: get_bool(f, "webSearch", "features", d.features.web_search)?,
|
||||
meals: get_bool(f, "meals", "features", d.features.meals)?,
|
||||
collect: get_bool(f, "collect", "features", d.features.collect)?,
|
||||
}
|
||||
},
|
||||
mcp_servers: parse_mcp_servers(raw)?,
|
||||
collect: parse_collect(raw, &d.collect)?,
|
||||
})
|
||||
}
|
||||
|
||||
/// A collection target's name: 1-32 letters, digits, `-` or `_` in any script. It names the
|
||||
/// target's folder, so a Windows device name (`con`, `nul`, `com1`, ...) is refused.
|
||||
pub fn valid_collect_name(s: &str) -> bool {
|
||||
let n = s.chars().count();
|
||||
let lower = s.to_ascii_lowercase();
|
||||
let device = ["con", "prn", "aux", "nul"].contains(&lower.as_str())
|
||||
|| (lower.len() == 4 && (lower.starts_with("com") || lower.starts_with("lpt"))
|
||||
&& lower.as_bytes()[3].is_ascii_digit());
|
||||
(1..=32).contains(&n) && s.chars().all(|c| c.is_alphanumeric() || c == '-' || c == '_') && !device
|
||||
}
|
||||
|
||||
fn get_uint(m: Option<&Map<String, Value>>, key: &str, path: &str, default: u32, min: u32, max: u32)
|
||||
-> Result<u32, SettingsError> {
|
||||
match m.and_then(|m| m.get(key)) {
|
||||
None => Ok(default),
|
||||
Some(v) => match v.as_u64() {
|
||||
Some(n) if (min as u64..=max as u64).contains(&n) => Ok(n as u32),
|
||||
_ => err(format!("{path}.{key} must be an integer {min}-{max}")),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_collect(raw: &Map<String, Value>, d: &CollectSettings) -> Result<CollectSettings, SettingsError> {
|
||||
let c = section(raw, "collect")?;
|
||||
let keep_days = get_uint(c, "keepDays", "collect", d.keep_days, 1, 365)?;
|
||||
let list = match c.and_then(|m| m.get("targets")) {
|
||||
None => return Ok(CollectSettings { keep_days, targets: vec![] }),
|
||||
Some(Value::Array(a)) => a,
|
||||
Some(_) => return err("collect.targets must be a list"),
|
||||
};
|
||||
let mut targets: Vec<CollectTarget> = Vec::new();
|
||||
for (i, v) in list.iter().enumerate() {
|
||||
let path = format!("collect.targets[{i}]");
|
||||
let Value::Object(m) = v else { return err(format!("{path} must be an object")) };
|
||||
let name = get_str(Some(m), "name", &path, "")?.trim().to_owned();
|
||||
if !valid_collect_name(&name) {
|
||||
return err(format!("{path}.name must be 1-32 letters, digits, - or _ (not a device name like con)"));
|
||||
}
|
||||
if targets.iter().any(|t| t.name == name) {
|
||||
return err(format!("{path}.name is listed twice"));
|
||||
}
|
||||
let kind = get_str(Some(m), "kind", &path, "")?;
|
||||
let value = get_str(Some(m), "value", &path, "")?.trim().to_owned();
|
||||
let ok = match kind.as_str() {
|
||||
"site" | "rss" => (value.starts_with("http://") || value.starts_with("https://")) && value.len() <= 2000,
|
||||
"search" => (1..=200).contains(&value.chars().count()),
|
||||
_ => return err(format!("{path}.kind must be site, search or rss")),
|
||||
};
|
||||
if !ok {
|
||||
return err(format!("{path}.value must be an http(s) address (site, rss) or 1-200 characters of words (search)"));
|
||||
}
|
||||
targets.push(CollectTarget {
|
||||
name,
|
||||
kind,
|
||||
value,
|
||||
every_minutes: get_uint(Some(m), "everyMinutes", &path, 60, 15, 10080)?,
|
||||
enabled: get_bool(Some(m), "enabled", &path, true)?,
|
||||
});
|
||||
}
|
||||
Ok(CollectSettings { keep_days, targets })
|
||||
}
|
||||
|
||||
/// An MCP server name: 1-32 letters, digits, `-` or `_`, without `__` (the brain names a server's
|
||||
/// tools `<server>__<tool>`).
|
||||
fn valid_mcp_name(s: &str) -> bool {
|
||||
@@ -718,13 +803,13 @@ mod tests {
|
||||
#[test]
|
||||
fn features_and_mcp_servers_are_read_and_checked() {
|
||||
let s = parse_settings(&json!({}));
|
||||
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true });
|
||||
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true, collect: true });
|
||||
let s = parse_settings(&json!({
|
||||
"features": {"weather": false, "meals": false},
|
||||
"mcpServers": [{"name": "brave", "command": "npx", "args": ["-y", "brave"], "env": ["BRAVE_API_KEY", " "]},
|
||||
{"name": "time-2", "command": "uvx", "enabled": false}]
|
||||
})).unwrap();
|
||||
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false });
|
||||
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false, collect: true });
|
||||
assert_eq!(s.mcp_servers, vec![
|
||||
McpServerSettings { name: "brave".into(), command: "npx".into(), args: vec!["-y".into(), "brave".into()],
|
||||
env: vec!["BRAVE_API_KEY".into()], enabled: true },
|
||||
@@ -747,6 +832,49 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn collection_targets_are_read_and_checked() {
|
||||
let s = parse_settings(&json!({})).unwrap();
|
||||
assert_eq!(s.collect, CollectSettings { keep_days: 30, targets: vec![] });
|
||||
let s = parse_settings(&json!({"features": {"collect": false}, "collect": {"keepDays": 7, "targets": [
|
||||
{"name": "뉴스_1", "kind": "rss", "value": " https://feed.example/rss "},
|
||||
{"name": "rust", "kind": "search", "value": "rust release", "everyMinutes": 1440, "enabled": false}]}})).unwrap();
|
||||
assert!(!s.features.collect);
|
||||
assert_eq!(s.collect, CollectSettings { keep_days: 7, targets: vec![
|
||||
CollectTarget { name: "뉴스_1".into(), kind: "rss".into(), value: "https://feed.example/rss".into(),
|
||||
every_minutes: 60, enabled: true },
|
||||
CollectTarget { name: "rust".into(), kind: "search".into(), value: "rust release".into(),
|
||||
every_minutes: 1440, enabled: false },
|
||||
]});
|
||||
let t = |extra: Value| {
|
||||
let mut m = json!({"name": "a", "kind": "site", "value": "https://a.example"});
|
||||
m.as_object_mut().unwrap().extend(extra.as_object().unwrap().clone());
|
||||
json!({"collect": {"targets": [m]}})
|
||||
};
|
||||
let cases = [
|
||||
(json!({"collect": {"keepDays": 0}}), "collect.keepDays"),
|
||||
(json!({"collect": {"keepDays": 366}}), "collect.keepDays"),
|
||||
(json!({"collect": {"targets": {}}}), "collect.targets"),
|
||||
(t(json!({"name": ""})), "collect.targets[0].name"),
|
||||
(t(json!({"name": "a/b"})), "collect.targets[0].name"),
|
||||
(t(json!({"name": "../x"})), "collect.targets[0].name"),
|
||||
(t(json!({"name": "CON"})), "collect.targets[0].name"),
|
||||
(t(json!({"name": "com1"})), "collect.targets[0].name"),
|
||||
(t(json!({"kind": "video"})), "collect.targets[0].kind"),
|
||||
(t(json!({"value": "ftp://a.example"})), "collect.targets[0].value"),
|
||||
(t(json!({"kind": "search", "value": " "})), "collect.targets[0].value"),
|
||||
(t(json!({"everyMinutes": 14})), "collect.targets[0].everyMinutes"),
|
||||
(t(json!({"everyMinutes": 10081})), "collect.targets[0].everyMinutes"),
|
||||
(json!({"collect": {"targets": [{"name": "a", "kind": "rss", "value": "https://a"},
|
||||
{"name": "a", "kind": "rss", "value": "https://b"}]}}), "collect.targets[1].name"),
|
||||
];
|
||||
for (raw, key) in cases {
|
||||
let e = parse_settings(&raw).unwrap_err();
|
||||
assert!(e.0.contains(key), "{raw} -> {e}");
|
||||
}
|
||||
assert!(valid_collect_name("console") && valid_collect_name("com") && valid_collect_name("날씨-서울"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn null_ids_clear_them() {
|
||||
let s = parse_settings(&json!({"discord": {"guildId": null, "voiceChannelId": null}})).unwrap();
|
||||
|
||||
@@ -20,6 +20,7 @@ pub enum Screen {
|
||||
Memory,
|
||||
Model,
|
||||
Features,
|
||||
Collect,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, serde::Serialize)]
|
||||
@@ -45,6 +46,8 @@ pub enum Kind {
|
||||
ChatChannels,
|
||||
/// The MCP servers (their own rules are checked by `parse_settings`; values are secrets).
|
||||
McpServers,
|
||||
/// The background collection targets (their own rules are checked by `parse_settings`).
|
||||
CollectTargets,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, serde::Serialize)]
|
||||
@@ -119,7 +122,11 @@ pub fn fields() -> Vec<Field> {
|
||||
f("features.weather", Features, Bool, json!(true), "날씨"),
|
||||
f("features.webSearch", Features, Bool, json!(true), "웹 검색"),
|
||||
f("features.meals", Features, Bool, json!(true), "식단 기록"),
|
||||
f("features.collect", Features, Bool, json!(true), "백그라운드 수집"),
|
||||
f("mcpServers", Features, McpServers, json!([]), "MCP 서버"),
|
||||
// Background collection (admin.spec.md, "Background collection"): no targets by default.
|
||||
f("collect.targets", Collect, CollectTargets, json!([]), "수집 대상"),
|
||||
f("collect.keepDays", Collect, Int { min: 1, max: 365 }, json!(30), "보관 기간(일)"),
|
||||
]
|
||||
}
|
||||
|
||||
@@ -209,7 +216,7 @@ fn check(field: &Field, v: &Value) -> Result<(), String> {
|
||||
Kind::IdList => v.as_array().is_some_and(|a| a.iter().all(is_id)),
|
||||
Kind::IdNumberMap => v.as_object().is_some_and(|m| m.iter()
|
||||
.all(|(k, n)| is_id(&Value::String(k.clone())) && n.as_i64().is_some())),
|
||||
Kind::ChatChannels | Kind::McpServers => v.is_array(),
|
||||
Kind::ChatChannels | Kind::McpServers | Kind::CollectTargets => v.is_array(),
|
||||
};
|
||||
if ok { Ok(()) } else { Err(format!("{} has a value it cannot take", field.path)) }
|
||||
}
|
||||
@@ -319,6 +326,7 @@ mod tests {
|
||||
(json!({"discord.commandName": "Big Name"}), "discord.commandName"),
|
||||
// A chat channel without a valid ID.
|
||||
(json!({"chatChannels": [{"id": "1"}]}), "chatChannels[0].id"),
|
||||
(json!({"collect.targets": [{"name": "nul", "kind": "rss", "value": "https://a"}]}), "collect.targets[0].name"),
|
||||
] {
|
||||
let e = apply(&p, &changes(c.clone()).into_iter().chain([("call.earcon".into(), json!(false))]).collect())
|
||||
.unwrap_err();
|
||||
|
||||
@@ -123,6 +123,8 @@ function editor(f, value) {
|
||||
return chatChannelsEditor(f, value);
|
||||
case "mcpServers":
|
||||
return mcpServersEditor(f, value);
|
||||
case "collectTargets":
|
||||
return collectTargetsEditor(f, value);
|
||||
default:
|
||||
el = h("span", { class: "muted" }, JSON.stringify(value));
|
||||
read = () => value;
|
||||
@@ -243,6 +245,67 @@ function mcpServersEditor(f, value) {
|
||||
list, h("button", { class: "btn", onclick: () => addServer() }, "서버 추가")), read, wide: true };
|
||||
}
|
||||
|
||||
// The background collection targets: one row each (admin.spec.md, "Background collection").
|
||||
const KIND = { site: "사이트", search: "검색어", rss: "RSS" };
|
||||
function collectTargetsEditor(f, value) {
|
||||
const body = h("tbody");
|
||||
const addRow = (t = {}) => {
|
||||
const kind = h("select", { "data-k": "kind" },
|
||||
Object.entries(KIND).map(([k, label]) => h("option", { value: k, selected: (t.kind ?? "site") === k }, label)));
|
||||
const val = h("input", { type: "text", value: t.value ?? "", "data-k": "value" });
|
||||
const hint = () => (val.placeholder = kind.value === "search" ? "검색어" : "https:// 주소");
|
||||
kind.addEventListener("change", hint);
|
||||
hint();
|
||||
const on = h("input", { type: "checkbox", "data-k": "enabled" });
|
||||
on.checked = t.enabled ?? true;
|
||||
const tr = h("tr", {},
|
||||
h("td", {}, h("input", { type: "text", value: t.name ?? "", placeholder: "이름 (글자, 숫자, - _)", "data-k": "name" })),
|
||||
h("td", {}, kind), h("td", {}, val),
|
||||
h("td", {}, h("input", { type: "number", min: 15, max: 10080, value: t.everyMinutes ?? 60, "data-k": "everyMinutes" })),
|
||||
h("td", {}, on),
|
||||
h("td", {}, h("button", { class: "btn", onclick: () => tr.remove() }, "삭제")));
|
||||
body.append(tr);
|
||||
};
|
||||
(value || []).forEach(addRow);
|
||||
const read = () => [...body.rows].map((tr) => {
|
||||
const k = (key) => tr.querySelector(`[data-k="${key}"]`);
|
||||
const every = Number(k("everyMinutes").value);
|
||||
if (!Number.isInteger(every)) throw new Error(`${f.label}: 주기는 분 단위 정수`);
|
||||
return { name: k("name").value.trim(), kind: k("kind").value, value: k("value").value.trim(),
|
||||
everyMinutes: every, enabled: k("enabled").checked };
|
||||
});
|
||||
const table = h("table", {}, h("thead", {}, h("tr", {}, ["이름", "종류", "주소 또는 검색어", "주기(분)", "켜기", ""]
|
||||
.map((x) => h("th", {}, x)))), body);
|
||||
return { el: h("div", {}, h("div", { class: "hint" }, "기본 수집 대상은 없습니다. 모은 내용은 데이터 폴더의 collect 폴더에만 저장됩니다."),
|
||||
table, h("button", { class: "btn", onclick: () => addRow() }, "대상 추가")), read, wide: true };
|
||||
}
|
||||
|
||||
/** The newest collected items, filtered by target; web content, so shown only as plain text. */
|
||||
async function collectResults() {
|
||||
const pick = h("select", {}, h("option", { value: "" }, "모든 대상"));
|
||||
const list = h("div");
|
||||
const load = async () => {
|
||||
try {
|
||||
const items = await api("collect_results", { target: pick.value || null, limit: 100 });
|
||||
if (!pick.value) {
|
||||
for (const name of new Set(items.map((i) => i.target)))
|
||||
if (![...pick.options].some((o) => o.value === name)) pick.append(h("option", { value: name }, name));
|
||||
}
|
||||
list.replaceChildren(items.length ? h("table", {}, h("tbody", {}, items.map((i) => h("tr", {},
|
||||
h("td", { class: "small muted" }, (i.at || "").replace("T", " ").slice(0, 16)),
|
||||
h("td", { class: "small" }, i.target),
|
||||
h("td", {}, h("div", {}, i.title || i.url || ""),
|
||||
i.url ? h("div", { class: "small muted" }, i.url) : null,
|
||||
i.text ? h("div", { class: "small" }, i.text.length > 200 ? i.text.slice(0, 200) + "…" : i.text) : null)))))
|
||||
: h("div", { class: "muted" }, "아직 모은 내용이 없습니다."));
|
||||
} catch (e) { fail(e); }
|
||||
};
|
||||
pick.addEventListener("change", load);
|
||||
load();
|
||||
return h("div", { class: "card" }, h("h2", {}, "수집 결과"),
|
||||
h("div", { class: "field" }, h("label", {}, "대상"), pick, h("button", { class: "btn", onclick: load }, "새로 고침")), list);
|
||||
}
|
||||
|
||||
async function settingsScreen(screen, title, extra) {
|
||||
const [all, view] = await Promise.all([fields(), api("settings_view")]);
|
||||
const list = all.filter((f) => f.screen === screen);
|
||||
@@ -476,9 +539,11 @@ async function show(name) {
|
||||
try {
|
||||
if (name === "dashboard") { screen.replaceChildren(); await refresh(); ticker = setInterval(refresh, 1000); return; }
|
||||
if (name === "logs") { const l = logsScreen(); screen.replaceChildren(l.view); ticker = setInterval(l.tick, 1000); return; }
|
||||
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능" };
|
||||
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능",
|
||||
collect: "백그라운드 수집" };
|
||||
const extra = name === "bot" ? await tokenCard() : null;
|
||||
const view = await settingsScreen(name, titles[name], extra);
|
||||
if (name === "collect") view.append(await collectResults());
|
||||
if (my === gen) screen.replaceChildren(view);
|
||||
} catch (e) { fail(e); }
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
<button data-screen="memory" class="nav">대화 기억</button>
|
||||
<button data-screen="model" class="nav">모델</button>
|
||||
<button data-screen="features" class="nav">기능</button>
|
||||
<button data-screen="collect" class="nav">백그라운드 수집</button>
|
||||
<button data-screen="logs" class="nav">로그</button>
|
||||
<div class="side-foot" id="version"></div>
|
||||
</nav>
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -67,6 +67,13 @@ def main() -> int:
|
||||
brain = JarvisBrain(language=args.tool_language or None, reply_language=args.reply_language or None,
|
||||
mcp_env=mcp_env)
|
||||
|
||||
# Background collection runs apart from the voice and the brain (voice_worker.spec.md).
|
||||
collector = None
|
||||
if os.environ.get("JARVIS_COLLECT_PATH"):
|
||||
from .collect import Collector, load_config
|
||||
collector = Collector(load_config(os.environ["JARVIS_COLLECT_PATH"]))
|
||||
collector.start()
|
||||
|
||||
async def run() -> None:
|
||||
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain)
|
||||
server = await worker.serve(args.host, args.port)
|
||||
@@ -81,6 +88,8 @@ def main() -> int:
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
if collector:
|
||||
collector.stop()
|
||||
if tts:
|
||||
tts.close()
|
||||
if history:
|
||||
|
||||
241
src/voice_worker/collect.py
Normal file
241
src/voice_worker/collect.py
Normal file
@@ -0,0 +1,241 @@
|
||||
"""Background collection of sites, search words and RSS feeds (voice_worker.spec.md, "Background
|
||||
collection"). Results go only to day files in the data folder; logs carry target names and counts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
import urllib.parse
|
||||
import xml.etree.ElementTree as ET
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional
|
||||
|
||||
from .log import debug_log, notice
|
||||
|
||||
MAX_BYTES = 1024 * 1024
|
||||
TIMEOUT_S = 15
|
||||
MAX_REDIRECTS = 5
|
||||
SITE_CHARS = 20_000
|
||||
SUMMARY_CHARS = 2_000
|
||||
SEEN_KEEP = 1000
|
||||
USER_AGENT = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/120.0.0.0 Safari/537.36")
|
||||
_DAY_FILE = re.compile(r"^(\d{4}-\d{2}-\d{2})\.jsonl$")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Target:
|
||||
name: str
|
||||
kind: str
|
||||
value: str
|
||||
every_s: float
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
dir: Path
|
||||
keep_days: int
|
||||
targets: list
|
||||
|
||||
|
||||
def load_config(path) -> Config:
|
||||
raw = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
return Config(Path(raw["dir"]), int(raw.get("keepDays", 30)),
|
||||
[Target(t["name"], t["kind"], t["value"], float(t["everyMinutes"]) * 60) for t in raw.get("targets", [])])
|
||||
|
||||
|
||||
def safe_get(url: str) -> bytes:
|
||||
"""GET a public http(s) address: every redirect checked again, at most ``MAX_BYTES``."""
|
||||
import requests
|
||||
from jarvis.tools.builtin.web_search import _is_public_url
|
||||
|
||||
for _ in range(MAX_REDIRECTS + 1):
|
||||
if not _is_public_url(url):
|
||||
raise ValueError("not a public address")
|
||||
with requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=TIMEOUT_S,
|
||||
allow_redirects=False, stream=True) as r:
|
||||
if r.is_redirect or r.is_permanent_redirect:
|
||||
url = urllib.parse.urljoin(url, r.headers.get("Location", ""))
|
||||
continue
|
||||
r.raise_for_status()
|
||||
if r.status_code != 200: # e.g. DuckDuckGo's 202 challenge
|
||||
raise ValueError(f"status {r.status_code}")
|
||||
body = b""
|
||||
for chunk in r.iter_content(64 * 1024):
|
||||
body += chunk
|
||||
if len(body) > MAX_BYTES:
|
||||
raise ValueError("response too large")
|
||||
return body
|
||||
raise ValueError("too many redirects")
|
||||
|
||||
|
||||
def _text(html) -> str:
|
||||
from bs4 import BeautifulSoup
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
for tag in soup(["script", "style", "noscript"]):
|
||||
tag.decompose()
|
||||
return " ".join(soup.get_text(" ").split())
|
||||
|
||||
|
||||
def _site(body: bytes) -> list:
|
||||
from bs4 import BeautifulSoup
|
||||
soup = BeautifulSoup(body, "html.parser")
|
||||
title = soup.title.get_text().strip() if soup.title else ""
|
||||
for tag in soup(["script", "style", "noscript", "title"]):
|
||||
tag.decompose()
|
||||
text = "\n".join(line for line in (" ".join(l.split()) for l in soup.get_text("\n").splitlines()) if line)
|
||||
return [{"title": title, "text": text[:SITE_CHARS]}]
|
||||
|
||||
|
||||
def _local(tag: str) -> str:
|
||||
return tag.rsplit("}", 1)[-1]
|
||||
|
||||
|
||||
def _feed(body: bytes) -> list:
|
||||
"""RSS 2.0 items or Atom entries, each with an ``id`` to tell new ones."""
|
||||
out = []
|
||||
for el in ET.fromstring(body).iter():
|
||||
if _local(el.tag) not in ("item", "entry"):
|
||||
continue
|
||||
f = {_local(c.tag): c for c in el}
|
||||
link = (f["link"].get("href") or (f["link"].text or "")).strip() if "link" in f else ""
|
||||
get = lambda *keys: next(((f[k].text or "").strip() for k in keys if k in f), "")
|
||||
item = {"title": get("title"), "url": link, "published": get("pubDate", "published", "updated"),
|
||||
"text": _text(get("description", "summary", "content"))[:SUMMARY_CHARS]}
|
||||
item["id"] = get("guid", "id") or link or item["title"]
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
def search_url(words: str) -> str:
|
||||
return "https://lite.duckduckgo.com/lite/?q=" + urllib.parse.quote_plus(words)
|
||||
|
||||
|
||||
class SearchBlocked(Exception):
|
||||
"""DuckDuckGo served its bot challenge instead of results (web_search.spec.md)."""
|
||||
|
||||
|
||||
def _search(body: bytes) -> list:
|
||||
from bs4 import BeautifulSoup
|
||||
if b"anomaly-modal" in body or b"anomaly.js" in body:
|
||||
raise SearchBlocked()
|
||||
out = []
|
||||
for a in BeautifulSoup(body, "html.parser").find_all("a", href=True):
|
||||
href, title = a["href"], " ".join(a.get_text().split())
|
||||
if href.startswith("//duckduckgo.com/l/"):
|
||||
href = urllib.parse.parse_qs(urllib.parse.urlparse(href).query).get("uddg", [""])[0]
|
||||
host = urllib.parse.urlparse(href).hostname or ""
|
||||
if href.startswith("http") and len(title) > 10 and not host.endswith("duckduckgo.com"):
|
||||
out.append({"title": title, "url": href, "id": href})
|
||||
return out[:10]
|
||||
|
||||
|
||||
class Collector:
|
||||
def __init__(self, cfg: Config, fetch: Callable[[str], bytes] = safe_get, check_every: float = 30.0):
|
||||
self.cfg = cfg
|
||||
self.fetch = fetch
|
||||
self.check_every = check_every
|
||||
self._stop = threading.Event()
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
|
||||
# -- state ----------------------------------------------------------------------------------
|
||||
def _folder(self, t: Target) -> Path:
|
||||
return self.cfg.dir / t.name
|
||||
|
||||
def _state(self, t: Target) -> dict:
|
||||
try:
|
||||
return json.loads((self._folder(t) / "state.json").read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
def _save_state(self, t: Target, state: dict) -> None:
|
||||
f = self._folder(t) / "state.json"
|
||||
f.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = f.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(state, ensure_ascii=False), encoding="utf-8")
|
||||
tmp.replace(f)
|
||||
|
||||
def _keep(self, t: Target, new: list) -> None:
|
||||
if not new:
|
||||
return
|
||||
f = self._folder(t) / f"{date.today()}.jsonl"
|
||||
f.parent.mkdir(parents=True, exist_ok=True)
|
||||
at = datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
with f.open("a", encoding="utf-8") as out:
|
||||
for item in new:
|
||||
out.write(json.dumps({"at": at, **item}, ensure_ascii=False) + "\n")
|
||||
|
||||
# -- one round ------------------------------------------------------------------------------
|
||||
def _prune(self) -> None:
|
||||
oldest = date.today() - timedelta(days=self.cfg.keep_days)
|
||||
if not self.cfg.dir.is_dir():
|
||||
return
|
||||
for folder in self.cfg.dir.iterdir():
|
||||
if not folder.is_dir():
|
||||
continue
|
||||
for f in folder.iterdir():
|
||||
m = _DAY_FILE.match(f.name)
|
||||
if m and date.fromisoformat(m.group(1)) < oldest:
|
||||
f.unlink(missing_ok=True)
|
||||
|
||||
def _run(self, t: Target, state: dict) -> int:
|
||||
if t.kind == "site":
|
||||
[page] = _site(self.fetch(t.value))
|
||||
digest = hashlib.sha256(page["text"].encode()).hexdigest()
|
||||
if digest == state.get("hash"):
|
||||
return 0
|
||||
state["hash"] = digest
|
||||
self._keep(t, [{**page, "url": t.value}])
|
||||
return 1
|
||||
found = _feed(self.fetch(t.value)) if t.kind == "rss" else _search(self.fetch(search_url(t.value)))
|
||||
seen = state.get("seen", [])
|
||||
new = [i for i in found if i["id"] not in seen]
|
||||
state["seen"] = (seen + [i["id"] for i in new])[-SEEN_KEEP:]
|
||||
self._keep(t, [{k: v for k, v in i.items() if k != "id"} for i in new])
|
||||
return len(new)
|
||||
|
||||
def run_due(self, now: Optional[float] = None) -> None:
|
||||
now = time.time() if now is None else now
|
||||
self._prune()
|
||||
for t in self.cfg.targets:
|
||||
if self._stop.is_set():
|
||||
return
|
||||
state = self._state(t)
|
||||
if now - state.get("last", float("-inf")) < t.every_s:
|
||||
continue
|
||||
state["last"] = now
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
n = self._run(t, state)
|
||||
notice(f"collect {t.name}: {n} new", "voice_worker")
|
||||
except Exception as e: # noqa: BLE001 - a broken target waits for its next interval
|
||||
notice(f"Warning: collect {t.name} failed: {type(e).__name__}", "voice_worker")
|
||||
self._save_state(t, state)
|
||||
debug_log(f"collect {t.name} kind={t.kind} ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
|
||||
|
||||
# -- thread ---------------------------------------------------------------------------------
|
||||
def start(self) -> None:
|
||||
def loop():
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
self.run_due()
|
||||
except Exception as e: # noqa: BLE001 - e.g. the folder became unwritable
|
||||
notice(f"Warning: collect round failed: {type(e).__name__}", "voice_worker")
|
||||
self._stop.wait(self.check_every)
|
||||
|
||||
self._thread = threading.Thread(target=loop, name="collect", daemon=True)
|
||||
self._thread.start()
|
||||
debug_log(f"collect started: {len(self.cfg.targets)} target(s)", "voice_worker")
|
||||
|
||||
def stop(self, timeout: float = 5.0) -> None:
|
||||
self._stop.set()
|
||||
if self._thread:
|
||||
self._thread.join(timeout)
|
||||
|
||||
def running(self) -> bool:
|
||||
return bool(self._thread and self._thread.is_alive())
|
||||
@@ -317,6 +317,29 @@ which carry Discord's own time.
|
||||
`budget_tokens` (4000).
|
||||
- Logs carry IDs, counts and timings only, never record, question or answer text.
|
||||
|
||||
### Background collection (M6)
|
||||
|
||||
With `JARVIS_COLLECT_PATH` set, the worker collects on its own thread, apart from the voice and the
|
||||
brain (`app/src/admin.spec.md`, "Background collection"). The file is
|
||||
`{"dir": "<data>/collect", "keepDays": 30, "targets": [{"name", "kind", "value", "everyMinutes"}]}`;
|
||||
the app writes only enabled targets. Nothing is collected without targets.
|
||||
|
||||
- A target runs when `everyMinutes` passed since its last run (also across restarts; the last run is
|
||||
kept in `<dir>/<name>/state.json`), one target at a time, checked every 30 s. A failed run counts
|
||||
as a run, so a broken site is tried again only at its next interval.
|
||||
- Kinds: `site` keeps the page's title and text (at most 20,000 characters) when it differs from the
|
||||
last kept one; `rss` (RSS or Atom) keeps each new item (title, link, time, summary of at most 2,000
|
||||
characters, new by its id or link); `search` keeps each new web search result (title and link) for
|
||||
the words, from DuckDuckGo's lite page, which needs no key (its bot challenge is a failed run, not "no results").
|
||||
- Every request goes through the web search's address check (`web_search.spec.md`): only public
|
||||
http(s) addresses, every redirect checked again, at most 1 MB, 15 s. The collected text is web
|
||||
content: it goes only to `<dir>/<name>/<YYYY-MM-DD>.jsonl` (one JSON object per line: `at`, then
|
||||
`title`, `url`, `text` as the kind has them), never to the brain or a log.
|
||||
- Day files older than `keepDays` are deleted in every folder of `dir` at each check, also of
|
||||
targets no longer listed.
|
||||
- Each run is a `notice` line (category `voice_worker`): the target's name and how many new items, or
|
||||
a `Warning` with the error's type. Never a URL, words or collected text.
|
||||
|
||||
### Errors
|
||||
|
||||
`{"type": "error", "code": "...", "message": "...", "id": "..."}`. Codes: `bad_message` (unknown type,
|
||||
|
||||
188
tests/test_voice_worker_collect.py
Normal file
188
tests/test_voice_worker_collect.py
Normal file
@@ -0,0 +1,188 @@
|
||||
"""Background collection (voice_worker.spec.md, "Background collection").
|
||||
|
||||
The collector runs against a fake fetcher (the real one refuses loopback addresses, which is
|
||||
checked on its own). Behaviours: each kind keeps only what is new, a target runs again only after
|
||||
its interval (also across a restart), a failure counts as a run and logs no URL, old day files go,
|
||||
and nothing collected reaches the log.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from voice_worker.collect import Collector, load_config, safe_get
|
||||
|
||||
RSS = """<?xml version="1.0"?><rss version="2.0"><channel><title>feed</title>
|
||||
<item><title>First</title><link>https://example.com/1</link><guid>id-1</guid>
|
||||
<pubDate>Fri, 13 Nov 2026 10:00:00 +0900</pubDate><description><p>one <b>bold</b></p></description></item>
|
||||
{more}
|
||||
</channel></rss>"""
|
||||
SECOND = """<item><title>Second</title><link>https://example.com/2</link><guid>id-2</guid></item>"""
|
||||
|
||||
ATOM = """<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom"><title>a</title>
|
||||
<entry><title>Atom one</title><id>tag:a,1</id><link href="https://example.org/a1"/>
|
||||
<updated>2026-11-13T01:00:00Z</updated><summary>sum</summary></entry></feed>"""
|
||||
|
||||
DDG = """<html><body><a href="https://duckduckgo.com/settings">Settings</a>
|
||||
<a href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fnews.example%2Fa&rut=x">A first result title here</a>
|
||||
<a href="https://news.example/b">Another result title here</a>{more}</body></html>"""
|
||||
|
||||
PAGE = "<html><head><title>Status</title><script>var x=1;</script></head><body><p>Version {v} is out</p></body></html>"
|
||||
|
||||
|
||||
class Fake:
|
||||
"""url -> bytes (or an exception to raise); records what was asked."""
|
||||
|
||||
def __init__(self, pages):
|
||||
self.pages = pages
|
||||
self.asked = []
|
||||
|
||||
def __call__(self, url):
|
||||
self.asked.append(url)
|
||||
v = self.pages[url]
|
||||
if isinstance(v, Exception):
|
||||
raise v
|
||||
return v.encode() if isinstance(v, str) else v
|
||||
|
||||
|
||||
def config(tmp_path, targets, keep_days=30):
|
||||
p = tmp_path / "collect.json"
|
||||
p.write_text(json.dumps({"dir": str(tmp_path / "collect"), "keepDays": keep_days, "targets": targets}))
|
||||
return load_config(p)
|
||||
|
||||
|
||||
def items(tmp_path, name):
|
||||
out = []
|
||||
for f in sorted((tmp_path / "collect" / name).glob("*.jsonl")):
|
||||
out += [json.loads(line) for line in f.read_text(encoding="utf-8").splitlines()]
|
||||
return out
|
||||
|
||||
|
||||
def lines(capfd):
|
||||
return capfd.readouterr().err
|
||||
|
||||
|
||||
def test_rss_keeps_each_new_item_once(tmp_path, capfd):
|
||||
fake = Fake({"https://feed.example/rss": RSS.format(more="")})
|
||||
cfg = config(tmp_path, [{"name": "뉴스", "kind": "rss", "value": "https://feed.example/rss", "everyMinutes": 15}])
|
||||
c = Collector(cfg, fetch=fake)
|
||||
c.run_due(now=1000.0)
|
||||
got = items(tmp_path, "뉴스")
|
||||
assert [i["title"] for i in got] == ["First"]
|
||||
assert got[0]["url"] == "https://example.com/1"
|
||||
assert got[0]["text"] == "one bold"
|
||||
assert got[0]["published"] == "Fri, 13 Nov 2026 10:00:00 +0900"
|
||||
fake.pages["https://feed.example/rss"] = RSS.format(more=SECOND)
|
||||
c.run_due(now=1000.0 + 15 * 60)
|
||||
assert [i["title"] for i in items(tmp_path, "뉴스")] == ["First", "Second"]
|
||||
out = lines(capfd)
|
||||
assert "collect 뉴스: 1 new" in out
|
||||
assert "feed.example" not in out and "First" not in out
|
||||
|
||||
|
||||
def test_atom_feeds_are_read_too(tmp_path):
|
||||
fake = Fake({"https://a.example/atom": ATOM})
|
||||
c = Collector(config(tmp_path, [{"name": "a", "kind": "rss", "value": "https://a.example/atom", "everyMinutes": 60}]),
|
||||
fetch=fake)
|
||||
c.run_due(now=0.0)
|
||||
[item] = items(tmp_path, "a")
|
||||
assert (item["title"], item["url"], item["text"]) == ("Atom one", "https://example.org/a1", "sum")
|
||||
|
||||
|
||||
def test_search_keeps_new_results_for_the_words(tmp_path):
|
||||
fake = Fake({})
|
||||
url = "https://lite.duckduckgo.com/lite/?q=rust+release"
|
||||
fake.pages[url] = DDG.format(more="")
|
||||
c = Collector(config(tmp_path, [{"name": "s", "kind": "search", "value": "rust release", "everyMinutes": 60}]),
|
||||
fetch=fake)
|
||||
c.run_due(now=0.0)
|
||||
assert [(i["title"], i["url"]) for i in items(tmp_path, "s")] == [
|
||||
("A first result title here", "https://news.example/a"), ("Another result title here", "https://news.example/b")]
|
||||
fake.pages[url] = DDG.format(more='<a href="https://news.example/c">A third result title</a>')
|
||||
c.run_due(now=3600.0)
|
||||
assert [i["url"] for i in items(tmp_path, "s")][-1] == "https://news.example/c"
|
||||
assert len(items(tmp_path, "s")) == 3
|
||||
|
||||
|
||||
def test_a_search_challenge_page_is_a_failure_not_zero_results(tmp_path, capfd):
|
||||
url = "https://lite.duckduckgo.com/lite/?q=words"
|
||||
fake = Fake({url: '<html><div class="anomaly-modal"></div><script src="//duckduckgo.com/anomaly.js"></script></html>'})
|
||||
c = Collector(config(tmp_path, [{"name": "s", "kind": "search", "value": "words", "everyMinutes": 60}]), fetch=fake)
|
||||
c.run_due(now=0.0)
|
||||
assert "Warning: collect s failed: SearchBlocked" in lines(capfd)
|
||||
assert not list((tmp_path / "collect").glob("s/*.jsonl"))
|
||||
|
||||
|
||||
def test_a_site_is_kept_only_when_it_changed(tmp_path):
|
||||
fake = Fake({"https://s.example/": PAGE.format(v=1)})
|
||||
c = Collector(config(tmp_path, [{"name": "site", "kind": "site", "value": "https://s.example/", "everyMinutes": 15}]),
|
||||
fetch=fake)
|
||||
c.run_due(now=0.0)
|
||||
c.run_due(now=900.0)
|
||||
got = items(tmp_path, "site")
|
||||
assert len(got) == 1 and got[0]["title"] == "Status" and got[0]["text"] == "Version 1 is out"
|
||||
fake.pages["https://s.example/"] = PAGE.format(v=2)
|
||||
c.run_due(now=1800.0)
|
||||
assert [i["text"] for i in items(tmp_path, "site")] == ["Version 1 is out", "Version 2 is out"]
|
||||
|
||||
|
||||
def test_a_target_runs_again_only_after_its_interval_also_after_a_restart(tmp_path):
|
||||
fake = Fake({"https://s.example/": PAGE.format(v=1)})
|
||||
targets = [{"name": "site", "kind": "site", "value": "https://s.example/", "everyMinutes": 60}]
|
||||
Collector(config(tmp_path, targets), fetch=fake).run_due(now=10_000.0)
|
||||
again = Collector(config(tmp_path, targets), fetch=fake)
|
||||
again.run_due(now=10_000.0 + 59 * 60)
|
||||
assert len(fake.asked) == 1
|
||||
again.run_due(now=10_000.0 + 60 * 60)
|
||||
assert len(fake.asked) == 2
|
||||
|
||||
|
||||
def test_a_failure_counts_as_a_run_and_logs_no_address(tmp_path, capfd):
|
||||
fake = Fake({"https://secret-site.example/": ConnectionError("down at https://secret-site.example/")})
|
||||
c = Collector(config(tmp_path, [{"name": "x", "kind": "site", "value": "https://secret-site.example/", "everyMinutes": 15}]),
|
||||
fetch=fake)
|
||||
c.run_due(now=0.0)
|
||||
c.run_due(now=60.0)
|
||||
assert len(fake.asked) == 1
|
||||
out = lines(capfd)
|
||||
assert "Warning: collect x failed: ConnectionError" in out
|
||||
assert "secret-site" not in out
|
||||
|
||||
|
||||
def test_old_day_files_go_in_every_folder(tmp_path):
|
||||
cfg = config(tmp_path, [], keep_days=2)
|
||||
today = datetime.now().date()
|
||||
for name in ("listed", "removed"):
|
||||
d = tmp_path / "collect" / name
|
||||
d.mkdir(parents=True)
|
||||
for age in (0, 2, 3):
|
||||
(d / f"{today - timedelta(days=age)}.jsonl").write_text("{}\n")
|
||||
(d / "state.json").write_text("{}")
|
||||
Collector(cfg, fetch=Fake({})).run_due(now=time.time())
|
||||
for name in ("listed", "removed"):
|
||||
left = sorted(p.name for p in (tmp_path / "collect" / name).iterdir())
|
||||
assert left == sorted([f"{today - timedelta(days=2)}.jsonl", f"{today}.jsonl", "state.json"])
|
||||
|
||||
|
||||
def test_the_real_fetcher_refuses_addresses_that_are_not_public():
|
||||
for url in ("http://127.0.0.1:1/", "http://192.168.0.1/", "file:///etc/passwd", "http://localhost/"):
|
||||
with pytest.raises(ValueError):
|
||||
safe_get(url)
|
||||
|
||||
|
||||
def test_the_thread_collects_on_its_own_and_stops(tmp_path):
|
||||
fake = Fake({"https://s.example/": PAGE.format(v=1)})
|
||||
c = Collector(config(tmp_path, [{"name": "site", "kind": "site", "value": "https://s.example/", "everyMinutes": 15}]),
|
||||
fetch=fake, check_every=0.05)
|
||||
c.start()
|
||||
deadline = time.monotonic() + 5
|
||||
while not list((tmp_path / "collect").glob("site/*.jsonl")):
|
||||
assert time.monotonic() < deadline
|
||||
time.sleep(0.02)
|
||||
c.stop()
|
||||
assert not c.running()
|
||||
Reference in New Issue
Block a user