feat(app): background collection frame for sites, search words and RSS feeds

Targets (none by default), interval and keep days on a new 백그라운드 수집 screen with the
results; the worker collects on its own thread through the web search's address check and
writes only to the data folder's collect/ (logs carry target names and counts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-13 21:29:49 +09:00
parent d531eb0ffc
commit b94e9dc112
15 changed files with 861 additions and 14 deletions

View File

@@ -38,9 +38,9 @@ that is not absolute, or is a file, is refused with a message.
`config/settings.json` keeps the format of `discord_core.spec.md` and `conversation.spec.md`.
- `fields()` describes every key the screens edit: its dotted path (`call.mode`), screen
(`bot`, `channels`, `chat`, `memory`, `model`, `features`), kind (bool, integer with a range, number with a
(`bot`, `channels`, `chat`, `memory`, `model`, `features`, `collect`), kind (bool, integer with a range, number with a
range, text, text list, choice with its values, Discord ID, Discord ID list, ID to number map,
chat channel list, MCP server list, free object), its default and a Korean label. The screens are built from this
chat channel list, MCP server list, collection target list, free object), its default and a Korean label. The screens are built from this
list, so a key is added in one place.
- `view(dir)` returns every field's current value: the file's value, or the default when the file
does not set it. A file that does not parse is reported, not overwritten.
@@ -59,10 +59,11 @@ that is not absolute, or is a file, is refused with a message.
manages the MCP servers it connects.
- `features` (new section): `weather` (`getWeather`), `webSearch` (`webSearch`, `fetchWebPage`) and
`meals` (`logMeal`, `fetchMeals`, `deleteMeal`), each on by default. The tools of a feature that
`meals` (`logMeal`, `fetchMeals`, `deleteMeal`), each on by default; `collect` (background
collection, below; no brain tool), also on by default. The tools of a feature that
is off go into the brain's `disabled_tools`: they are neither offered nor run
(`voice_worker.spec.md`, "Tools"). The other features of the 기능 screen (browser control, PC
control, screen viewing, background collection) come with their milestones (M6, M7).
control, screen viewing) come with their milestones (M6, M7).
- `mcpServers` (new list): each `{name, command, args, env, enabled}`. `name` is 1-32 letters,
digits, `-` or `_` without `__` (the brain names a server's tools `<name>__<tool>`), not listed
twice; `command` is not empty; `args` a list of strings; `env` the names of the environment
@@ -79,6 +80,26 @@ manages the MCP servers it connects.
also without debug logging. MCP servers are programs the PC runs; the screen says to add only
trusted ones.
## Background collection (`app/src/collect.rs`)
`docs/v2/desktop-app.md`, "백그라운드 수집", and `docs/v2/pc-features.md`. The frame for collecting
sites, search words and RSS feeds on a schedule; there are **no targets by default**.
- `collect` (new section): `keepDays` (1-365, default 30) and `targets` (default empty), each
`{name, kind, value, everyMinutes, enabled}`. `name` is 1-32 letters, digits, `-` or `_` (any
script; it names the target's folder, so not a Windows device name like `con` or `nul`), not
listed twice; `kind` is `site`, `search` or `rss`; `value` is an http(s) address for `site` and
`rss` (at most 2,000 characters) and the words for `search` (1-200 characters);
`everyMinutes` 15-10080 (default 60); `enabled` on by default.
- On start, with `features.collect` on and at least one enabled target, the app writes
`config/collect.json` (the folder `collect/`, `keepDays` and the enabled targets) and passes it to
the worker as `JARVIS_COLLECT_PATH`; otherwise it removes that file and passes nothing. The worker
collects while the services run (`voice_worker.spec.md`, "Background collection"); results stay
in `collect/<name>/<YYYY-MM-DD>.jsonl` in the data folder only.
- `results(data, target, limit)` reads them newest first (the newest day file first, its last line
first), each item with its `target`; a name that is not a valid target name is refused, so no
other folder is read. Lines that do not parse are skipped.
## Secrets (`app/src/secrets.rs`)
`config/secrets.dat` holds a JSON object of named secrets (`discordToken`, the MCP servers'
@@ -284,6 +305,7 @@ back as Korean messages for the screen. The Tauri shell passes the screens' `inv
| `settings_fields` / `settings_view` / `settings_apply` `{changes}` | the settings editor above; `restartNeeded` when the services run; `sync` with the chat channels sent to Discord (below) |
| `service_start` / `service_stop` / `service_restart` / `dashboard` | the controller and its snapshot; start and restart run in the background, stop also cancels a start |
| `discord_channel_info` `{id}` / `channel_sync_status` | a channel's Discord name and topic / each synced chat channel's last result (above) |
| `collect_results` `{target, limit}` | collected items newest first (`target` null: every target; `limit` 1-500, default 100) |
| `mcp_env_status` / `mcp_env_set` `{server, name, value}` | which MCP environment values are stored (`{server: {NAME: bool}}`, never the values) / store one (an empty value removes it); only for a saved server and name |
| `install_status` / `install_start` / `install_cancel` | the install steps and their state, bytes and last line / run what is not done in the background / stop it |
| `logs_query` `{tab, level, find, after, limit}` / `logs_error_report` / `logs_export` `{from, to}` | the log hub; the zip goes to `logs/` and its path comes back |
@@ -309,13 +331,16 @@ Plain HTML, CSS and an ES module, no build step, dark theme after `DESIGN.md`. A
Strata's last line, and stop in the header; the header with start, restart and stop is rebuilt
only when the state changes, so a click is never lost to the refresh. Every screen change starts
a new generation: an answer that arrives after the user moved to another screen is dropped.
- **Settings screens** (봇, 채널·호출, 채팅 채널, 대화 기억, 모델, 기능): built from `settings_fields`; only
- **Settings screens** (봇, 채널·호출, 채팅 채널, 대화 기억, 모델, 기능, 백그라운드 수집): built from `settings_fields`; only
changed fields are sent; the server and voice channel are pickers (choosing another server clears
the voice channel and lists only that server's channels); the chat channels are a table
with their switches; the bot screen also changes the token; the 기능 screen has the feature
switches and one card per MCP server (name, command, arguments and environment variable names
one per line, on/off, delete), and for a saved server a password box per environment variable
that stores its value on its own.
that stores its value on its own. The 백그라운드 수집 screen has the targets (a table: name, kind,
address or words, interval in minutes, on/off, delete), the keep days, and the results (the
newest 100, filtered by target, each with its time, title, link and the start of its text shown
as plain text).
- **Logs**: tab, level and text filters, pause, live follow (lines after the last seen), "최근 오류
복사" (to the clipboard; over 2,000 characters it suggests the zip), and the zip export by days.

View File

@@ -281,6 +281,11 @@ impl AdminApi {
Ok(json!({"set": !value.is_empty(),
"restartNeeded": self.controller()?.state() == crate::control::State::Running}))
}
"collect_results" => {
let target = args.get("target").and_then(Value::as_str).filter(|t| !t.is_empty());
let limit = args.get("limit").and_then(Value::as_u64).unwrap_or(100).clamp(1, 500) as usize;
Ok(json!(crate::collect::results(self.dir()?.path(), target, limit)?))
}
"channel_sync_status" => Ok(Value::Object(self.syncer()?.status().into_iter()
.map(|(id, s)| (id.to_string(), json!(s))).collect())),
"discord_channel_info" => {
@@ -392,6 +397,22 @@ mod tests {
assert!(a.call("nope", json!({})).await.is_err());
}
#[tokio::test]
async fn collection_targets_are_saved_and_their_results_read_from_the_data_folder() {
let (a, base) = api("api-collect");
let data = base.join("d");
a.call("folder_choose", json!({"path": data.to_string_lossy()})).await.unwrap();
let targets = json!([{"name": "뉴스", "kind": "rss", "value": "https://feed.example/rss", "everyMinutes": 30}]);
a.call("settings_apply", json!({"changes": {"collect.targets": targets, "collect.keepDays": 7}})).await.unwrap();
assert_eq!(a.call("settings_view", json!({})).await.unwrap()["collect.targets"], targets);
let day = data.join("collect").join("뉴스");
std::fs::create_dir_all(&day).unwrap();
std::fs::write(day.join("2026-11-13.jsonl"), "{\"at\": \"2026-11-13T09:00:00+09:00\", \"title\": \"t\"}\n").unwrap();
let r = a.call("collect_results", json!({"target": null})).await.unwrap();
assert_eq!(r, json!([{"at": "2026-11-13T09:00:00+09:00", "title": "t", "target": "뉴스"}]));
assert!(a.call("collect_results", json!({"target": "../config"})).await.is_err());
}
#[tokio::test]
async fn mcp_values_are_stored_encrypted_shown_only_as_set_masked_in_logs_and_go_with_their_server() {
let (a, base) = api("api-mcp");

131
app/src/collect.rs Normal file
View File

@@ -0,0 +1,131 @@
//! Background collection (`app/src/admin.spec.md`, "Background collection"): the worker's
//! collection file, and the collected items for the 백그라운드 수집 screen.
use std::path::{Path, PathBuf};
use serde_json::{json, Value};
use crate::settings::{valid_collect_name, Settings};
/// The worker's environment variable with the collection file (`voice_worker/collect.py`).
pub const COLLECT_ENV_VAR: &str = "JARVIS_COLLECT_PATH";
/// Write `config/collect.json` with the enabled targets and return it; with the feature off or no
/// enabled target, remove it and return None (the worker then collects nothing).
pub fn write_config(data: &Path, settings: &Settings) -> std::io::Result<Option<PathBuf>> {
let file = data.join("config").join("collect.json");
let targets: Vec<Value> = settings.collect.targets.iter().filter(|t| t.enabled)
.map(|t| json!({"name": t.name, "kind": t.kind, "value": t.value, "everyMinutes": t.every_minutes}))
.collect();
if !settings.features.collect || targets.is_empty() {
match std::fs::remove_file(&file) {
Err(e) if e.kind() != std::io::ErrorKind::NotFound => return Err(e),
_ => return Ok(None),
}
}
let cfg = json!({"dir": data.join("collect").to_string_lossy(), "keepDays": settings.collect.keep_days,
"targets": targets});
let mut text = serde_json::to_string_pretty(&cfg).unwrap();
text.push('\n');
crate::datadir::write_atomic(&file, text.as_bytes())?;
Ok(Some(file))
}
/// The day files of one target folder, newest first.
fn day_files(folder: &Path) -> Vec<PathBuf> {
let mut files: Vec<PathBuf> = std::fs::read_dir(folder).into_iter().flatten().flatten().map(|e| e.path())
.filter(|p| p.file_name().and_then(|n| n.to_str()).is_some_and(|n| n.len() == 16 && n.ends_with(".jsonl")))
.collect();
files.sort();
files.reverse();
files
}
/// Collected items newest first, each with its `target`; `target` None reads every target folder.
pub fn results(data: &Path, target: Option<&str>, limit: usize) -> Result<Vec<Value>, String> {
let root = data.join("collect");
let names: Vec<String> = match target {
Some(t) if !valid_collect_name(t) => return Err(format!("not a collection target name: {t}")),
Some(t) => vec![t.to_owned()],
None => std::fs::read_dir(&root).into_iter().flatten().flatten()
.filter(|e| e.path().is_dir())
.filter_map(|e| e.file_name().to_str().map(str::to_owned))
.filter(|n| valid_collect_name(n))
.collect(),
};
let mut out: Vec<Value> = Vec::new();
for name in names {
let mut n = 0;
'files: for f in day_files(&root.join(&name)) {
let text = std::fs::read_to_string(&f).unwrap_or_default();
for line in text.lines().rev() {
if let Ok(Value::Object(mut item)) = serde_json::from_str::<Value>(line) {
item.insert("target".into(), Value::String(name.clone()));
out.push(Value::Object(item));
n += 1;
if n == limit {
break 'files;
}
}
}
}
}
// `at` is the local time with its offset in one format, so the text order is the time order.
out.sort_by(|a, b| b["at"].as_str().unwrap_or("").cmp(a["at"].as_str().unwrap_or("")));
out.truncate(limit);
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::datadir::temp_dir;
use crate::settings::parse_settings;
fn settings(v: Value) -> Settings {
parse_settings(&v).unwrap()
}
#[test]
fn the_worker_gets_only_enabled_targets_and_nothing_without_them() {
let data = temp_dir("collect-cfg");
let s = settings(json!({"collect": {"keepDays": 3, "targets": [
{"name": "a", "kind": "rss", "value": "https://a.example/rss", "everyMinutes": 30},
{"name": "b", "kind": "site", "value": "https://b.example", "enabled": false}]}}));
let file = write_config(&data, &s).unwrap().unwrap();
let cfg: Value = serde_json::from_str(&std::fs::read_to_string(&file).unwrap()).unwrap();
assert_eq!(cfg, json!({"dir": data.join("collect").to_string_lossy(), "keepDays": 3,
"targets": [{"name": "a", "kind": "rss", "value": "https://a.example/rss", "everyMinutes": 30}]}));
let mut off = s.clone();
off.features.collect = false;
assert_eq!(write_config(&data, &off).unwrap(), None);
assert!(!file.exists());
write_config(&data, &s).unwrap();
assert_eq!(write_config(&data, &settings(json!({}))).unwrap(), None);
assert!(!file.exists());
}
#[test]
fn results_come_newest_first_with_their_target_and_only_from_target_folders() {
let data = temp_dir("collect-res");
let put = |name: &str, day: &str, lines: &[&str]| {
let d = data.join("collect").join(name);
std::fs::create_dir_all(&d).unwrap();
std::fs::write(d.join(format!("{day}.jsonl")), lines.join("\n") + "\n").unwrap();
};
put("뉴스", "2026-11-12", &[r#"{"at": "2026-11-12T09:00:00+09:00", "title": "old"}"#]);
put("뉴스", "2026-11-13", &[r#"{"at": "2026-11-13T09:00:00+09:00", "title": "n1"}"#, "not json",
r#"{"at": "2026-11-13T10:00:00+09:00", "title": "n2"}"#]);
put("site", "2026-11-13", &[r#"{"at": "2026-11-13T09:30:00+09:00", "title": "s1"}"#]);
std::fs::write(data.join("collect").join("뉴스").join("state.json"), r#"{"at": "9999"}"#).unwrap();
let titles = |v: Vec<Value>| v.iter().map(|i| format!("{}:{}", i["target"].as_str().unwrap(),
i["title"].as_str().unwrap())).collect::<Vec<_>>();
assert_eq!(titles(results(&data, None, 100).unwrap()), ["뉴스:n2", "site:s1", "뉴스:n1", "뉴스:old"]);
assert_eq!(titles(results(&data, Some("뉴스"), 2).unwrap()), ["뉴스:n2", "뉴스:n1"]);
assert_eq!(titles(results(&data, None, 2).unwrap()), ["뉴스:n2", "site:s1"]);
assert!(results(&data, Some("missing"), 10).unwrap().is_empty());
assert!(results(&data, Some(".."), 10).is_err());
assert!(results(&data, Some("../config"), 10).is_err());
}
}

View File

@@ -100,9 +100,9 @@ mod tests {
#[test]
fn features_that_are_off_turn_their_tools_off() {
let all_on = FeatureSettings { weather: true, web_search: true, meals: true };
let all_on = FeatureSettings { weather: true, web_search: true, meals: true, collect: true };
assert!(disabled_tools(&all_on).is_empty());
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false });
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false, collect: false });
assert_eq!(off, vec!["getWeather", "logMeal", "fetchMeals", "deleteMeal"]);
}

View File

@@ -5,6 +5,7 @@ pub mod admin_api;
pub mod calls;
pub mod channel_sync;
pub mod chat;
pub mod collect;
pub mod control;
pub mod conversation;
pub mod dashboard;

View File

@@ -70,6 +70,12 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
}
wopts.env.insert("JARVIS_CONFIG_PATH".into(), brain_config.to_string_lossy().into_owned());
wopts.env.insert(crate::features::MCP_ENV_VAR.into(), mcp_env.to_string());
// Background collection: only with the feature on and enabled targets (admin.spec.md).
match crate::collect::write_config(data_dir, &settings) {
Ok(Some(f)) => { wopts.env.insert(crate::collect::COLLECT_ENV_VAR.into(), f.to_string_lossy().into_owned()); }
Ok(None) => {}
Err(e) => return Err(after_llm_stop(&llm, format!("could not write the collection settings: {e}")).await.into()),
}
let worker_process = match WorkerProcess::new(wopts) {
Ok(p) => p,
Err(e) => return Err(after_llm_stop(&llm, e.to_string()).await.into()),

View File

@@ -91,6 +91,25 @@ pub struct FeatureSettings {
pub weather: bool,
pub web_search: bool,
pub meals: bool,
/// Background collection (admin.spec.md, "Background collection").
pub collect: bool,
}
/// Background collection (admin.spec.md, "Background collection"); no targets by default.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CollectSettings {
pub keep_days: u32,
pub targets: Vec<CollectTarget>,
}
/// One thing to collect: `kind` is `site`, `search` or `rss`; `value` its address or words.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CollectTarget {
pub name: String,
pub kind: String,
pub value: String,
pub every_minutes: u32,
pub enabled: bool,
}
/// An MCP server the brain starts (stdio). The values of `env` are in the secrets, never here.
@@ -121,6 +140,7 @@ pub struct Settings {
pub llm: LlmSettings,
pub features: FeatureSettings,
pub mcp_servers: Vec<McpServerSettings>,
pub collect: CollectSettings,
}
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -164,8 +184,9 @@ impl Default for Settings {
response: PolicySettings::default(),
memory: Map::new(),
llm: LlmSettings { url: None, port: 8080, context: 32768 },
features: FeatureSettings { weather: true, web_search: true, meals: true },
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true },
mcp_servers: vec![],
collect: CollectSettings { keep_days: 30, targets: vec![] },
}
}
}
@@ -483,12 +504,76 @@ pub fn parse_settings(raw: &Value) -> Result<Settings, SettingsError> {
weather: get_bool(f, "weather", "features", d.features.weather)?,
web_search: get_bool(f, "webSearch", "features", d.features.web_search)?,
meals: get_bool(f, "meals", "features", d.features.meals)?,
collect: get_bool(f, "collect", "features", d.features.collect)?,
}
},
mcp_servers: parse_mcp_servers(raw)?,
collect: parse_collect(raw, &d.collect)?,
})
}
/// A collection target's name: 1-32 letters, digits, `-` or `_` in any script. It names the
/// target's folder, so a Windows device name (`con`, `nul`, `com1`, ...) is refused.
pub fn valid_collect_name(s: &str) -> bool {
let n = s.chars().count();
let lower = s.to_ascii_lowercase();
let device = ["con", "prn", "aux", "nul"].contains(&lower.as_str())
|| (lower.len() == 4 && (lower.starts_with("com") || lower.starts_with("lpt"))
&& lower.as_bytes()[3].is_ascii_digit());
(1..=32).contains(&n) && s.chars().all(|c| c.is_alphanumeric() || c == '-' || c == '_') && !device
}
fn get_uint(m: Option<&Map<String, Value>>, key: &str, path: &str, default: u32, min: u32, max: u32)
-> Result<u32, SettingsError> {
match m.and_then(|m| m.get(key)) {
None => Ok(default),
Some(v) => match v.as_u64() {
Some(n) if (min as u64..=max as u64).contains(&n) => Ok(n as u32),
_ => err(format!("{path}.{key} must be an integer {min}-{max}")),
},
}
}
fn parse_collect(raw: &Map<String, Value>, d: &CollectSettings) -> Result<CollectSettings, SettingsError> {
let c = section(raw, "collect")?;
let keep_days = get_uint(c, "keepDays", "collect", d.keep_days, 1, 365)?;
let list = match c.and_then(|m| m.get("targets")) {
None => return Ok(CollectSettings { keep_days, targets: vec![] }),
Some(Value::Array(a)) => a,
Some(_) => return err("collect.targets must be a list"),
};
let mut targets: Vec<CollectTarget> = Vec::new();
for (i, v) in list.iter().enumerate() {
let path = format!("collect.targets[{i}]");
let Value::Object(m) = v else { return err(format!("{path} must be an object")) };
let name = get_str(Some(m), "name", &path, "")?.trim().to_owned();
if !valid_collect_name(&name) {
return err(format!("{path}.name must be 1-32 letters, digits, - or _ (not a device name like con)"));
}
if targets.iter().any(|t| t.name == name) {
return err(format!("{path}.name is listed twice"));
}
let kind = get_str(Some(m), "kind", &path, "")?;
let value = get_str(Some(m), "value", &path, "")?.trim().to_owned();
let ok = match kind.as_str() {
"site" | "rss" => (value.starts_with("http://") || value.starts_with("https://")) && value.len() <= 2000,
"search" => (1..=200).contains(&value.chars().count()),
_ => return err(format!("{path}.kind must be site, search or rss")),
};
if !ok {
return err(format!("{path}.value must be an http(s) address (site, rss) or 1-200 characters of words (search)"));
}
targets.push(CollectTarget {
name,
kind,
value,
every_minutes: get_uint(Some(m), "everyMinutes", &path, 60, 15, 10080)?,
enabled: get_bool(Some(m), "enabled", &path, true)?,
});
}
Ok(CollectSettings { keep_days, targets })
}
/// An MCP server name: 1-32 letters, digits, `-` or `_`, without `__` (the brain names a server's
/// tools `<server>__<tool>`).
fn valid_mcp_name(s: &str) -> bool {
@@ -718,13 +803,13 @@ mod tests {
#[test]
fn features_and_mcp_servers_are_read_and_checked() {
let s = parse_settings(&json!({}));
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true });
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true, collect: true });
let s = parse_settings(&json!({
"features": {"weather": false, "meals": false},
"mcpServers": [{"name": "brave", "command": "npx", "args": ["-y", "brave"], "env": ["BRAVE_API_KEY", " "]},
{"name": "time-2", "command": "uvx", "enabled": false}]
})).unwrap();
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false });
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false, collect: true });
assert_eq!(s.mcp_servers, vec![
McpServerSettings { name: "brave".into(), command: "npx".into(), args: vec!["-y".into(), "brave".into()],
env: vec!["BRAVE_API_KEY".into()], enabled: true },
@@ -747,6 +832,49 @@ mod tests {
}
}
#[test]
fn collection_targets_are_read_and_checked() {
let s = parse_settings(&json!({})).unwrap();
assert_eq!(s.collect, CollectSettings { keep_days: 30, targets: vec![] });
let s = parse_settings(&json!({"features": {"collect": false}, "collect": {"keepDays": 7, "targets": [
{"name": "뉴스_1", "kind": "rss", "value": " https://feed.example/rss "},
{"name": "rust", "kind": "search", "value": "rust release", "everyMinutes": 1440, "enabled": false}]}})).unwrap();
assert!(!s.features.collect);
assert_eq!(s.collect, CollectSettings { keep_days: 7, targets: vec![
CollectTarget { name: "뉴스_1".into(), kind: "rss".into(), value: "https://feed.example/rss".into(),
every_minutes: 60, enabled: true },
CollectTarget { name: "rust".into(), kind: "search".into(), value: "rust release".into(),
every_minutes: 1440, enabled: false },
]});
let t = |extra: Value| {
let mut m = json!({"name": "a", "kind": "site", "value": "https://a.example"});
m.as_object_mut().unwrap().extend(extra.as_object().unwrap().clone());
json!({"collect": {"targets": [m]}})
};
let cases = [
(json!({"collect": {"keepDays": 0}}), "collect.keepDays"),
(json!({"collect": {"keepDays": 366}}), "collect.keepDays"),
(json!({"collect": {"targets": {}}}), "collect.targets"),
(t(json!({"name": ""})), "collect.targets[0].name"),
(t(json!({"name": "a/b"})), "collect.targets[0].name"),
(t(json!({"name": "../x"})), "collect.targets[0].name"),
(t(json!({"name": "CON"})), "collect.targets[0].name"),
(t(json!({"name": "com1"})), "collect.targets[0].name"),
(t(json!({"kind": "video"})), "collect.targets[0].kind"),
(t(json!({"value": "ftp://a.example"})), "collect.targets[0].value"),
(t(json!({"kind": "search", "value": " "})), "collect.targets[0].value"),
(t(json!({"everyMinutes": 14})), "collect.targets[0].everyMinutes"),
(t(json!({"everyMinutes": 10081})), "collect.targets[0].everyMinutes"),
(json!({"collect": {"targets": [{"name": "a", "kind": "rss", "value": "https://a"},
{"name": "a", "kind": "rss", "value": "https://b"}]}}), "collect.targets[1].name"),
];
for (raw, key) in cases {
let e = parse_settings(&raw).unwrap_err();
assert!(e.0.contains(key), "{raw} -> {e}");
}
assert!(valid_collect_name("console") && valid_collect_name("com") && valid_collect_name("날씨-서울"));
}
#[test]
fn null_ids_clear_them() {
let s = parse_settings(&json!({"discord": {"guildId": null, "voiceChannelId": null}})).unwrap();

View File

@@ -20,6 +20,7 @@ pub enum Screen {
Memory,
Model,
Features,
Collect,
}
#[derive(Debug, Clone, PartialEq, serde::Serialize)]
@@ -45,6 +46,8 @@ pub enum Kind {
ChatChannels,
/// The MCP servers (their own rules are checked by `parse_settings`; values are secrets).
McpServers,
/// The background collection targets (their own rules are checked by `parse_settings`).
CollectTargets,
}
#[derive(Debug, Clone, PartialEq, serde::Serialize)]
@@ -119,7 +122,11 @@ pub fn fields() -> Vec<Field> {
f("features.weather", Features, Bool, json!(true), "날씨"),
f("features.webSearch", Features, Bool, json!(true), "웹 검색"),
f("features.meals", Features, Bool, json!(true), "식단 기록"),
f("features.collect", Features, Bool, json!(true), "백그라운드 수집"),
f("mcpServers", Features, McpServers, json!([]), "MCP 서버"),
// Background collection (admin.spec.md, "Background collection"): no targets by default.
f("collect.targets", Collect, CollectTargets, json!([]), "수집 대상"),
f("collect.keepDays", Collect, Int { min: 1, max: 365 }, json!(30), "보관 기간(일)"),
]
}
@@ -209,7 +216,7 @@ fn check(field: &Field, v: &Value) -> Result<(), String> {
Kind::IdList => v.as_array().is_some_and(|a| a.iter().all(is_id)),
Kind::IdNumberMap => v.as_object().is_some_and(|m| m.iter()
.all(|(k, n)| is_id(&Value::String(k.clone())) && n.as_i64().is_some())),
Kind::ChatChannels | Kind::McpServers => v.is_array(),
Kind::ChatChannels | Kind::McpServers | Kind::CollectTargets => v.is_array(),
};
if ok { Ok(()) } else { Err(format!("{} has a value it cannot take", field.path)) }
}
@@ -319,6 +326,7 @@ mod tests {
(json!({"discord.commandName": "Big Name"}), "discord.commandName"),
// A chat channel without a valid ID.
(json!({"chatChannels": [{"id": "1"}]}), "chatChannels[0].id"),
(json!({"collect.targets": [{"name": "nul", "kind": "rss", "value": "https://a"}]}), "collect.targets[0].name"),
] {
let e = apply(&p, &changes(c.clone()).into_iter().chain([("call.earcon".into(), json!(false))]).collect())
.unwrap_err();

View File

@@ -123,6 +123,8 @@ function editor(f, value) {
return chatChannelsEditor(f, value);
case "mcpServers":
return mcpServersEditor(f, value);
case "collectTargets":
return collectTargetsEditor(f, value);
default:
el = h("span", { class: "muted" }, JSON.stringify(value));
read = () => value;
@@ -243,6 +245,67 @@ function mcpServersEditor(f, value) {
list, h("button", { class: "btn", onclick: () => addServer() }, "서버 추가")), read, wide: true };
}
// The background collection targets: one row each (admin.spec.md, "Background collection").
const KIND = { site: "사이트", search: "검색어", rss: "RSS" };
function collectTargetsEditor(f, value) {
const body = h("tbody");
const addRow = (t = {}) => {
const kind = h("select", { "data-k": "kind" },
Object.entries(KIND).map(([k, label]) => h("option", { value: k, selected: (t.kind ?? "site") === k }, label)));
const val = h("input", { type: "text", value: t.value ?? "", "data-k": "value" });
const hint = () => (val.placeholder = kind.value === "search" ? "검색어" : "https:// 주소");
kind.addEventListener("change", hint);
hint();
const on = h("input", { type: "checkbox", "data-k": "enabled" });
on.checked = t.enabled ?? true;
const tr = h("tr", {},
h("td", {}, h("input", { type: "text", value: t.name ?? "", placeholder: "이름 (글자, 숫자, - _)", "data-k": "name" })),
h("td", {}, kind), h("td", {}, val),
h("td", {}, h("input", { type: "number", min: 15, max: 10080, value: t.everyMinutes ?? 60, "data-k": "everyMinutes" })),
h("td", {}, on),
h("td", {}, h("button", { class: "btn", onclick: () => tr.remove() }, "삭제")));
body.append(tr);
};
(value || []).forEach(addRow);
const read = () => [...body.rows].map((tr) => {
const k = (key) => tr.querySelector(`[data-k="${key}"]`);
const every = Number(k("everyMinutes").value);
if (!Number.isInteger(every)) throw new Error(`${f.label}: 주기는 분 단위 정수`);
return { name: k("name").value.trim(), kind: k("kind").value, value: k("value").value.trim(),
everyMinutes: every, enabled: k("enabled").checked };
});
const table = h("table", {}, h("thead", {}, h("tr", {}, ["이름", "종류", "주소 또는 검색어", "주기(분)", "켜기", ""]
.map((x) => h("th", {}, x)))), body);
return { el: h("div", {}, h("div", { class: "hint" }, "기본 수집 대상은 없습니다. 모은 내용은 데이터 폴더의 collect 폴더에만 저장됩니다."),
table, h("button", { class: "btn", onclick: () => addRow() }, "대상 추가")), read, wide: true };
}
/** The newest collected items, filtered by target; web content, so shown only as plain text. */
async function collectResults() {
const pick = h("select", {}, h("option", { value: "" }, "모든 대상"));
const list = h("div");
const load = async () => {
try {
const items = await api("collect_results", { target: pick.value || null, limit: 100 });
if (!pick.value) {
for (const name of new Set(items.map((i) => i.target)))
if (![...pick.options].some((o) => o.value === name)) pick.append(h("option", { value: name }, name));
}
list.replaceChildren(items.length ? h("table", {}, h("tbody", {}, items.map((i) => h("tr", {},
h("td", { class: "small muted" }, (i.at || "").replace("T", " ").slice(0, 16)),
h("td", { class: "small" }, i.target),
h("td", {}, h("div", {}, i.title || i.url || ""),
i.url ? h("div", { class: "small muted" }, i.url) : null,
i.text ? h("div", { class: "small" }, i.text.length > 200 ? i.text.slice(0, 200) + "…" : i.text) : null)))))
: h("div", { class: "muted" }, "아직 모은 내용이 없습니다."));
} catch (e) { fail(e); }
};
pick.addEventListener("change", load);
load();
return h("div", { class: "card" }, h("h2", {}, "수집 결과"),
h("div", { class: "field" }, h("label", {}, "대상"), pick, h("button", { class: "btn", onclick: load }, "새로 고침")), list);
}
async function settingsScreen(screen, title, extra) {
const [all, view] = await Promise.all([fields(), api("settings_view")]);
const list = all.filter((f) => f.screen === screen);
@@ -476,9 +539,11 @@ async function show(name) {
try {
if (name === "dashboard") { screen.replaceChildren(); await refresh(); ticker = setInterval(refresh, 1000); return; }
if (name === "logs") { const l = logsScreen(); screen.replaceChildren(l.view); ticker = setInterval(l.tick, 1000); return; }
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능" };
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능",
collect: "백그라운드 수집" };
const extra = name === "bot" ? await tokenCard() : null;
const view = await settingsScreen(name, titles[name], extra);
if (name === "collect") view.append(await collectResults());
if (my === gen) screen.replaceChildren(view);
} catch (e) { fail(e); }
}

View File

@@ -18,6 +18,7 @@
<button data-screen="memory" class="nav">대화 기억</button>
<button data-screen="model" class="nav">모델</button>
<button data-screen="features" class="nav">기능</button>
<button data-screen="collect" class="nav">백그라운드 수집</button>
<button data-screen="logs" class="nav">로그</button>
<div class="side-foot" id="version"></div>
</nav>

File diff suppressed because one or more lines are too long

View File

@@ -67,6 +67,13 @@ def main() -> int:
brain = JarvisBrain(language=args.tool_language or None, reply_language=args.reply_language or None,
mcp_env=mcp_env)
# Background collection runs apart from the voice and the brain (voice_worker.spec.md).
collector = None
if os.environ.get("JARVIS_COLLECT_PATH"):
from .collect import Collector, load_config
collector = Collector(load_config(os.environ["JARVIS_COLLECT_PATH"]))
collector.start()
async def run() -> None:
worker = VoiceWorker(token, transcriber, vad.stream, tts, history=history, brain=brain)
server = await worker.serve(args.host, args.port)
@@ -81,6 +88,8 @@ def main() -> int:
except KeyboardInterrupt:
pass
finally:
if collector:
collector.stop()
if tts:
tts.close()
if history:

241
src/voice_worker/collect.py Normal file
View File

@@ -0,0 +1,241 @@
"""Background collection of sites, search words and RSS feeds (voice_worker.spec.md, "Background
collection"). Results go only to day files in the data folder; logs carry target names and counts."""
from __future__ import annotations
import hashlib
import json
import re
import threading
import time
import urllib.parse
import xml.etree.ElementTree as ET
from dataclasses import dataclass
from datetime import date, datetime, timedelta
from pathlib import Path
from typing import Callable, Optional
from .log import debug_log, notice
MAX_BYTES = 1024 * 1024
TIMEOUT_S = 15
MAX_REDIRECTS = 5
SITE_CHARS = 20_000
SUMMARY_CHARS = 2_000
SEEN_KEEP = 1000
USER_AGENT = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36")
_DAY_FILE = re.compile(r"^(\d{4}-\d{2}-\d{2})\.jsonl$")
@dataclass
class Target:
name: str
kind: str
value: str
every_s: float
@dataclass
class Config:
dir: Path
keep_days: int
targets: list
def load_config(path) -> Config:
raw = json.loads(Path(path).read_text(encoding="utf-8"))
return Config(Path(raw["dir"]), int(raw.get("keepDays", 30)),
[Target(t["name"], t["kind"], t["value"], float(t["everyMinutes"]) * 60) for t in raw.get("targets", [])])
def safe_get(url: str) -> bytes:
"""GET a public http(s) address: every redirect checked again, at most ``MAX_BYTES``."""
import requests
from jarvis.tools.builtin.web_search import _is_public_url
for _ in range(MAX_REDIRECTS + 1):
if not _is_public_url(url):
raise ValueError("not a public address")
with requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=TIMEOUT_S,
allow_redirects=False, stream=True) as r:
if r.is_redirect or r.is_permanent_redirect:
url = urllib.parse.urljoin(url, r.headers.get("Location", ""))
continue
r.raise_for_status()
if r.status_code != 200: # e.g. DuckDuckGo's 202 challenge
raise ValueError(f"status {r.status_code}")
body = b""
for chunk in r.iter_content(64 * 1024):
body += chunk
if len(body) > MAX_BYTES:
raise ValueError("response too large")
return body
raise ValueError("too many redirects")
def _text(html) -> str:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "noscript"]):
tag.decompose()
return " ".join(soup.get_text(" ").split())
def _site(body: bytes) -> list:
from bs4 import BeautifulSoup
soup = BeautifulSoup(body, "html.parser")
title = soup.title.get_text().strip() if soup.title else ""
for tag in soup(["script", "style", "noscript", "title"]):
tag.decompose()
text = "\n".join(line for line in (" ".join(l.split()) for l in soup.get_text("\n").splitlines()) if line)
return [{"title": title, "text": text[:SITE_CHARS]}]
def _local(tag: str) -> str:
return tag.rsplit("}", 1)[-1]
def _feed(body: bytes) -> list:
"""RSS 2.0 items or Atom entries, each with an ``id`` to tell new ones."""
out = []
for el in ET.fromstring(body).iter():
if _local(el.tag) not in ("item", "entry"):
continue
f = {_local(c.tag): c for c in el}
link = (f["link"].get("href") or (f["link"].text or "")).strip() if "link" in f else ""
get = lambda *keys: next(((f[k].text or "").strip() for k in keys if k in f), "")
item = {"title": get("title"), "url": link, "published": get("pubDate", "published", "updated"),
"text": _text(get("description", "summary", "content"))[:SUMMARY_CHARS]}
item["id"] = get("guid", "id") or link or item["title"]
out.append(item)
return out
def search_url(words: str) -> str:
return "https://lite.duckduckgo.com/lite/?q=" + urllib.parse.quote_plus(words)
class SearchBlocked(Exception):
"""DuckDuckGo served its bot challenge instead of results (web_search.spec.md)."""
def _search(body: bytes) -> list:
from bs4 import BeautifulSoup
if b"anomaly-modal" in body or b"anomaly.js" in body:
raise SearchBlocked()
out = []
for a in BeautifulSoup(body, "html.parser").find_all("a", href=True):
href, title = a["href"], " ".join(a.get_text().split())
if href.startswith("//duckduckgo.com/l/"):
href = urllib.parse.parse_qs(urllib.parse.urlparse(href).query).get("uddg", [""])[0]
host = urllib.parse.urlparse(href).hostname or ""
if href.startswith("http") and len(title) > 10 and not host.endswith("duckduckgo.com"):
out.append({"title": title, "url": href, "id": href})
return out[:10]
class Collector:
def __init__(self, cfg: Config, fetch: Callable[[str], bytes] = safe_get, check_every: float = 30.0):
self.cfg = cfg
self.fetch = fetch
self.check_every = check_every
self._stop = threading.Event()
self._thread: Optional[threading.Thread] = None
# -- state ----------------------------------------------------------------------------------
def _folder(self, t: Target) -> Path:
return self.cfg.dir / t.name
def _state(self, t: Target) -> dict:
try:
return json.loads((self._folder(t) / "state.json").read_text(encoding="utf-8"))
except (OSError, ValueError):
return {}
def _save_state(self, t: Target, state: dict) -> None:
f = self._folder(t) / "state.json"
f.parent.mkdir(parents=True, exist_ok=True)
tmp = f.with_suffix(".tmp")
tmp.write_text(json.dumps(state, ensure_ascii=False), encoding="utf-8")
tmp.replace(f)
def _keep(self, t: Target, new: list) -> None:
if not new:
return
f = self._folder(t) / f"{date.today()}.jsonl"
f.parent.mkdir(parents=True, exist_ok=True)
at = datetime.now().astimezone().isoformat(timespec="seconds")
with f.open("a", encoding="utf-8") as out:
for item in new:
out.write(json.dumps({"at": at, **item}, ensure_ascii=False) + "\n")
# -- one round ------------------------------------------------------------------------------
def _prune(self) -> None:
oldest = date.today() - timedelta(days=self.cfg.keep_days)
if not self.cfg.dir.is_dir():
return
for folder in self.cfg.dir.iterdir():
if not folder.is_dir():
continue
for f in folder.iterdir():
m = _DAY_FILE.match(f.name)
if m and date.fromisoformat(m.group(1)) < oldest:
f.unlink(missing_ok=True)
def _run(self, t: Target, state: dict) -> int:
if t.kind == "site":
[page] = _site(self.fetch(t.value))
digest = hashlib.sha256(page["text"].encode()).hexdigest()
if digest == state.get("hash"):
return 0
state["hash"] = digest
self._keep(t, [{**page, "url": t.value}])
return 1
found = _feed(self.fetch(t.value)) if t.kind == "rss" else _search(self.fetch(search_url(t.value)))
seen = state.get("seen", [])
new = [i for i in found if i["id"] not in seen]
state["seen"] = (seen + [i["id"] for i in new])[-SEEN_KEEP:]
self._keep(t, [{k: v for k, v in i.items() if k != "id"} for i in new])
return len(new)
def run_due(self, now: Optional[float] = None) -> None:
now = time.time() if now is None else now
self._prune()
for t in self.cfg.targets:
if self._stop.is_set():
return
state = self._state(t)
if now - state.get("last", float("-inf")) < t.every_s:
continue
state["last"] = now
t0 = time.perf_counter()
try:
n = self._run(t, state)
notice(f"collect {t.name}: {n} new", "voice_worker")
except Exception as e: # noqa: BLE001 - a broken target waits for its next interval
notice(f"Warning: collect {t.name} failed: {type(e).__name__}", "voice_worker")
self._save_state(t, state)
debug_log(f"collect {t.name} kind={t.kind} ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
# -- thread ---------------------------------------------------------------------------------
def start(self) -> None:
def loop():
while not self._stop.is_set():
try:
self.run_due()
except Exception as e: # noqa: BLE001 - e.g. the folder became unwritable
notice(f"Warning: collect round failed: {type(e).__name__}", "voice_worker")
self._stop.wait(self.check_every)
self._thread = threading.Thread(target=loop, name="collect", daemon=True)
self._thread.start()
debug_log(f"collect started: {len(self.cfg.targets)} target(s)", "voice_worker")
def stop(self, timeout: float = 5.0) -> None:
self._stop.set()
if self._thread:
self._thread.join(timeout)
def running(self) -> bool:
return bool(self._thread and self._thread.is_alive())

View File

@@ -317,6 +317,29 @@ which carry Discord's own time.
`budget_tokens` (4000).
- Logs carry IDs, counts and timings only, never record, question or answer text.
### Background collection (M6)
With `JARVIS_COLLECT_PATH` set, the worker collects on its own thread, apart from the voice and the
brain (`app/src/admin.spec.md`, "Background collection"). The file is
`{"dir": "<data>/collect", "keepDays": 30, "targets": [{"name", "kind", "value", "everyMinutes"}]}`;
the app writes only enabled targets. Nothing is collected without targets.
- A target runs when `everyMinutes` passed since its last run (also across restarts; the last run is
kept in `<dir>/<name>/state.json`), one target at a time, checked every 30 s. A failed run counts
as a run, so a broken site is tried again only at its next interval.
- Kinds: `site` keeps the page's title and text (at most 20,000 characters) when it differs from the
last kept one; `rss` (RSS or Atom) keeps each new item (title, link, time, summary of at most 2,000
characters, new by its id or link); `search` keeps each new web search result (title and link) for
the words, from DuckDuckGo's lite page, which needs no key (its bot challenge is a failed run, not "no results").
- Every request goes through the web search's address check (`web_search.spec.md`): only public
http(s) addresses, every redirect checked again, at most 1 MB, 15 s. The collected text is web
content: it goes only to `<dir>/<name>/<YYYY-MM-DD>.jsonl` (one JSON object per line: `at`, then
`title`, `url`, `text` as the kind has them), never to the brain or a log.
- Day files older than `keepDays` are deleted in every folder of `dir` at each check, also of
targets no longer listed.
- Each run is a `notice` line (category `voice_worker`): the target's name and how many new items, or
a `Warning` with the error's type. Never a URL, words or collected text.
### Errors
`{"type": "error", "code": "...", "message": "...", "id": "..."}`. Codes: `bad_message` (unknown type,

View File

@@ -0,0 +1,188 @@
"""Background collection (voice_worker.spec.md, "Background collection").
The collector runs against a fake fetcher (the real one refuses loopback addresses, which is
checked on its own). Behaviours: each kind keeps only what is new, a target runs again only after
its interval (also across a restart), a failure counts as a run and logs no URL, old day files go,
and nothing collected reaches the log.
"""
from __future__ import annotations
import json
import time
from datetime import datetime, timedelta
from pathlib import Path
import pytest
from voice_worker.collect import Collector, load_config, safe_get
RSS = """<?xml version="1.0"?><rss version="2.0"><channel><title>feed</title>
<item><title>First</title><link>https://example.com/1</link><guid>id-1</guid>
<pubDate>Fri, 13 Nov 2026 10:00:00 +0900</pubDate><description>&lt;p&gt;one &lt;b&gt;bold&lt;/b&gt;&lt;/p&gt;</description></item>
{more}
</channel></rss>"""
SECOND = """<item><title>Second</title><link>https://example.com/2</link><guid>id-2</guid></item>"""
ATOM = """<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom"><title>a</title>
<entry><title>Atom one</title><id>tag:a,1</id><link href="https://example.org/a1"/>
<updated>2026-11-13T01:00:00Z</updated><summary>sum</summary></entry></feed>"""
DDG = """<html><body><a href="https://duckduckgo.com/settings">Settings</a>
<a href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fnews.example%2Fa&amp;rut=x">A first result title here</a>
<a href="https://news.example/b">Another result title here</a>{more}</body></html>"""
PAGE = "<html><head><title>Status</title><script>var x=1;</script></head><body><p>Version {v} is out</p></body></html>"
class Fake:
"""url -> bytes (or an exception to raise); records what was asked."""
def __init__(self, pages):
self.pages = pages
self.asked = []
def __call__(self, url):
self.asked.append(url)
v = self.pages[url]
if isinstance(v, Exception):
raise v
return v.encode() if isinstance(v, str) else v
def config(tmp_path, targets, keep_days=30):
p = tmp_path / "collect.json"
p.write_text(json.dumps({"dir": str(tmp_path / "collect"), "keepDays": keep_days, "targets": targets}))
return load_config(p)
def items(tmp_path, name):
out = []
for f in sorted((tmp_path / "collect" / name).glob("*.jsonl")):
out += [json.loads(line) for line in f.read_text(encoding="utf-8").splitlines()]
return out
def lines(capfd):
return capfd.readouterr().err
def test_rss_keeps_each_new_item_once(tmp_path, capfd):
fake = Fake({"https://feed.example/rss": RSS.format(more="")})
cfg = config(tmp_path, [{"name": "뉴스", "kind": "rss", "value": "https://feed.example/rss", "everyMinutes": 15}])
c = Collector(cfg, fetch=fake)
c.run_due(now=1000.0)
got = items(tmp_path, "뉴스")
assert [i["title"] for i in got] == ["First"]
assert got[0]["url"] == "https://example.com/1"
assert got[0]["text"] == "one bold"
assert got[0]["published"] == "Fri, 13 Nov 2026 10:00:00 +0900"
fake.pages["https://feed.example/rss"] = RSS.format(more=SECOND)
c.run_due(now=1000.0 + 15 * 60)
assert [i["title"] for i in items(tmp_path, "뉴스")] == ["First", "Second"]
out = lines(capfd)
assert "collect 뉴스: 1 new" in out
assert "feed.example" not in out and "First" not in out
def test_atom_feeds_are_read_too(tmp_path):
fake = Fake({"https://a.example/atom": ATOM})
c = Collector(config(tmp_path, [{"name": "a", "kind": "rss", "value": "https://a.example/atom", "everyMinutes": 60}]),
fetch=fake)
c.run_due(now=0.0)
[item] = items(tmp_path, "a")
assert (item["title"], item["url"], item["text"]) == ("Atom one", "https://example.org/a1", "sum")
def test_search_keeps_new_results_for_the_words(tmp_path):
fake = Fake({})
url = "https://lite.duckduckgo.com/lite/?q=rust+release"
fake.pages[url] = DDG.format(more="")
c = Collector(config(tmp_path, [{"name": "s", "kind": "search", "value": "rust release", "everyMinutes": 60}]),
fetch=fake)
c.run_due(now=0.0)
assert [(i["title"], i["url"]) for i in items(tmp_path, "s")] == [
("A first result title here", "https://news.example/a"), ("Another result title here", "https://news.example/b")]
fake.pages[url] = DDG.format(more='<a href="https://news.example/c">A third result title</a>')
c.run_due(now=3600.0)
assert [i["url"] for i in items(tmp_path, "s")][-1] == "https://news.example/c"
assert len(items(tmp_path, "s")) == 3
def test_a_search_challenge_page_is_a_failure_not_zero_results(tmp_path, capfd):
url = "https://lite.duckduckgo.com/lite/?q=words"
fake = Fake({url: '<html><div class="anomaly-modal"></div><script src="//duckduckgo.com/anomaly.js"></script></html>'})
c = Collector(config(tmp_path, [{"name": "s", "kind": "search", "value": "words", "everyMinutes": 60}]), fetch=fake)
c.run_due(now=0.0)
assert "Warning: collect s failed: SearchBlocked" in lines(capfd)
assert not list((tmp_path / "collect").glob("s/*.jsonl"))
def test_a_site_is_kept_only_when_it_changed(tmp_path):
fake = Fake({"https://s.example/": PAGE.format(v=1)})
c = Collector(config(tmp_path, [{"name": "site", "kind": "site", "value": "https://s.example/", "everyMinutes": 15}]),
fetch=fake)
c.run_due(now=0.0)
c.run_due(now=900.0)
got = items(tmp_path, "site")
assert len(got) == 1 and got[0]["title"] == "Status" and got[0]["text"] == "Version 1 is out"
fake.pages["https://s.example/"] = PAGE.format(v=2)
c.run_due(now=1800.0)
assert [i["text"] for i in items(tmp_path, "site")] == ["Version 1 is out", "Version 2 is out"]
def test_a_target_runs_again_only_after_its_interval_also_after_a_restart(tmp_path):
fake = Fake({"https://s.example/": PAGE.format(v=1)})
targets = [{"name": "site", "kind": "site", "value": "https://s.example/", "everyMinutes": 60}]
Collector(config(tmp_path, targets), fetch=fake).run_due(now=10_000.0)
again = Collector(config(tmp_path, targets), fetch=fake)
again.run_due(now=10_000.0 + 59 * 60)
assert len(fake.asked) == 1
again.run_due(now=10_000.0 + 60 * 60)
assert len(fake.asked) == 2
def test_a_failure_counts_as_a_run_and_logs_no_address(tmp_path, capfd):
fake = Fake({"https://secret-site.example/": ConnectionError("down at https://secret-site.example/")})
c = Collector(config(tmp_path, [{"name": "x", "kind": "site", "value": "https://secret-site.example/", "everyMinutes": 15}]),
fetch=fake)
c.run_due(now=0.0)
c.run_due(now=60.0)
assert len(fake.asked) == 1
out = lines(capfd)
assert "Warning: collect x failed: ConnectionError" in out
assert "secret-site" not in out
def test_old_day_files_go_in_every_folder(tmp_path):
cfg = config(tmp_path, [], keep_days=2)
today = datetime.now().date()
for name in ("listed", "removed"):
d = tmp_path / "collect" / name
d.mkdir(parents=True)
for age in (0, 2, 3):
(d / f"{today - timedelta(days=age)}.jsonl").write_text("{}\n")
(d / "state.json").write_text("{}")
Collector(cfg, fetch=Fake({})).run_due(now=time.time())
for name in ("listed", "removed"):
left = sorted(p.name for p in (tmp_path / "collect" / name).iterdir())
assert left == sorted([f"{today - timedelta(days=2)}.jsonl", f"{today}.jsonl", "state.json"])
def test_the_real_fetcher_refuses_addresses_that_are_not_public():
for url in ("http://127.0.0.1:1/", "http://192.168.0.1/", "file:///etc/passwd", "http://localhost/"):
with pytest.raises(ValueError):
safe_get(url)
def test_the_thread_collects_on_its_own_and_stops(tmp_path):
fake = Fake({"https://s.example/": PAGE.format(v=1)})
c = Collector(config(tmp_path, [{"name": "site", "kind": "site", "value": "https://s.example/", "everyMinutes": 15}]),
fetch=fake, check_every=0.05)
c.start()
deadline = time.monotonic() + 5
while not list((tmp_path / "collect").glob("site/*.jsonl")):
assert time.monotonic() < deadline
time.sleep(0.02)
c.stop()
assert not c.running()