feat(app): the bot's own Chrome in the data folder, reaching only public addresses

- install step browser: Chrome for Testing 157 (pinned SHA-256) into runtime/chrome
- services start it headless with its own profile when features.browser is on; CDP on a free
  loopback port passed to the worker as JARVIS_BROWSER_CDP; a failure leaves the services running
- every Chrome connection goes through the app's loopback egress proxy: one lookup, public
  addresses only (403 otherwise, no host logged), QUIC off, WebRTC only through the proxy
- stop: Browser.close, then kill after 5 s; dashboard shows the browser's state
- the worker's runtime environment follows the settings step, not every step being installed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-13 22:01:04 +09:00
parent 69f1bf4fa9
commit 24d0798bae
12 changed files with 724 additions and 16 deletions

View File

@@ -60,10 +60,11 @@ manages the MCP servers it connects.
- `features` (new section): `weather` (`getWeather`), `webSearch` (`webSearch`, `fetchWebPage`) and
`meals` (`logMeal`, `fetchMeals`, `deleteMeal`), each on by default; `collect` (background
collection, below; no brain tool), also on by default. The tools of a feature that
collection, below; no brain tool) and `browser` (the bot browser, below "Bot browser"; its brain
tool comes with it), also on by default. The tools of a feature that
is off go into the brain's `disabled_tools`: they are neither offered nor run
(`voice_worker.spec.md`, "Tools"). The other features of the 기능 screen (browser control, PC
control, screen viewing) come with their milestones (M6, M7).
(`voice_worker.spec.md`, "Tools"). The other features of the 기능 screen (PC control, screen
viewing) come with their milestone (M7).
- `mcpServers` (new list): each `{name, command, args, env, enabled}`. `name` is 1-32 letters,
digits, `-` or `_` without `__` (the brain names a server's tools `<name>__<tool>`), not listed
twice; `command` is not empty; `args` a list of strings; `env` the names of the environment
@@ -183,6 +184,7 @@ reaches the screen.
who, why and how long ago, never the text.
- `system`: memory in use and total, and per NVIDIA GPU its use and VRAM (from `nvidia-smi`,
missing when there is none); sampled when the snapshot is taken, at most once a second.
- `browser`: the bot browser's `{state, note}` ("Bot browser").
- `llm`: the LLM's state (`off`, `starting`, `ready`, `failed`), whether the app runs it
(`managed`), and while starting Strata's last line and how far that line says it got (its
percentage, or `X of Y GB` for the model download; none on a line without progress), also before
@@ -206,6 +208,7 @@ on the game PC (`scripts/m0`). The steps, in order:
| `stt_model` | Whisper `large-v3-turbo` (faster-whisper) |
| `tts_model` | the NLTK data MeloTTS's English g2p reads, then the worker's own `melo_child.py` loads the Korean voice once (downloading it) and must say its warm-up sentence |
| `strata` | Strata's own installer (`install.ps1`, `install.sh` elsewhere) with Strata's state, engine and components under `runtime/strata` and no PATH change |
| `browser` | the bot's Chrome: Chrome for Testing 157.0.8103.78 (win64, linux64), downloaded with resume and checked against its pinned SHA-256, unpacked into `runtime/chrome` (below, "Bot browser") |
| `settings` | `voiceWorker` in the settings pointed at the worker venv's Python with `--melo-python <melo venv> --history memory/history.sqlite --brain` (other settings kept); the install is done only once this is written |
The LLM itself (Qwen3.6-35B-A3B, about 23 GB) is downloaded by Strata on its first start (below).
@@ -230,6 +233,37 @@ The LLM itself (Qwen3.6-35B-A3B, about 23 GB) is downloaded by Strata on its fir
## Bot browser (`app/src/browser.rs`)
`docs/v2/pc-features.md`, "백그라운드 크로미움으로 정보 수집" and "봇용 크롬창과 디스코드 화면공유". The
bot's own Chrome, in the data folder, driven over the Chrome DevTools Protocol (CDP). This part runs
it in the background (headless); the virtual monitor for screen sharing comes next (M6).
- `features.browser` (브라우저 제어, on by default). With it on and the `browser` step installed,
the services start the bot browser after the LLM and before the worker, and stop it after the
worker. Without it (off, or not installed) nothing starts. A browser that does not start is a
warning on the dashboard and in the `browser` log tab; the services run on without it.
- Chrome runs with its own profile `browser/profile` (`--user-data-dir`), headless, no first-run,
sync, component updates, translation or background networking, and CDP on a free loopback port
(`--remote-debugging-port=0`; the port is read from the profile's `DevToolsActivePort`, which is
removed before each start, and `/json/version` must answer within 30 s). The worker gets
`JARVIS_BROWSER_CDP` = `http://127.0.0.1:<port>`.
- **Egress proxy**: every Chrome connection goes through the app's proxy on a free loopback port
(`--proxy-server`, and `--proxy-bypass-list=<-loopback>` so not even loopback is direct; QUIC off,
WebRTC only through the proxy). The proxy takes `CONNECT host:port` (HTTPS, WebSocket) and plain
HTTP requests with an absolute `http://` address. It resolves the name once (at most 10 s), refuses
with 403 unless every address is public (not loopback, private, link-local, CGNAT, documentation,
benchmarking, multicast, reserved or unspecified, also inside IPv4-mapped IPv6), and connects
only to those addresses, so Chrome can never reach the PC or the local network. Plain HTTP goes
on with `Connection: close` and without proxy headers. A request it cannot read is a 400. It logs
no host or address, only that it refused one.
- Stop: CDP `Browser.close`, then up to 5 s for the process to end, else it and the processes it
started are killed. The services' stop ends the proxy too.
- Dashboard `browser`: `{state, note}` with `state` off, ready, failed (the note says why) or exited
(it ended on its own).
- Linux (development only): Ubuntu's AppArmor refuses Chrome's sandbox; `JARVIS_CHROME_NO_SANDBOX=1`
in the app's environment adds `--no-sandbox`. Windows always runs it sandboxed.
## LLM (`app/src/strata.rs`)
The brain answers with one model, Qwen3.6-35B-A3B UD-Q4_K_XL through Strata (M0). The services

425
app/src/browser.rs Normal file
View File

@@ -0,0 +1,425 @@
//! The bot browser (`app/src/admin.spec.md`, "Bot browser"): the bot's own Chrome in the data
//! folder, driven over CDP, and the egress proxy every Chrome connection goes through so it can
//! only reach public addresses.
use std::net::{IpAddr, SocketAddr};
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
const RESOLVE_TIMEOUT: Duration = Duration::from_secs(10);
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
const HEAD_TIMEOUT: Duration = Duration::from_secs(30);
const MAX_HEAD: usize = 64 * 1024;
const READY_TIMEOUT: Duration = Duration::from_secs(30);
const CLOSE_WAIT: Duration = Duration::from_secs(5);
/// Whether an address is on the public internet: not this PC, the local network or a special range.
pub fn is_public(ip: &IpAddr) -> bool {
match ip {
IpAddr::V4(v4) => {
let o = v4.octets();
!(v4.is_private() || v4.is_loopback() || v4.is_link_local() || v4.is_broadcast()
|| v4.is_documentation() || v4.is_unspecified() || v4.is_multicast()
|| o[0] == 0 // "this network"
|| (o[0] == 100 && (64..128).contains(&o[1])) // CGNAT 100.64/10
|| (o[0] == 192 && o[1] == 0 && o[2] == 0) // IETF 192.0.0/24
|| (o[0] == 198 && (18..20).contains(&o[1])) // benchmarking 198.18/15
|| o[0] >= 240) // reserved
}
IpAddr::V6(v6) => {
if let Some(v4) = v6.to_ipv4_mapped() {
return is_public(&IpAddr::V4(v4));
}
let s = v6.segments();
!(v6.is_loopback() || v6.is_unspecified() || v6.is_multicast()
|| (s[0] & 0xfe00) == 0xfc00 // unique local fc00::/7
|| (s[0] & 0xffc0) == 0xfe80 // link local fe80::/10
|| (s[0] == 0x2001 && s[1] == 0x0db8) // documentation
|| (s[0] == 0x0064 && s[1] == 0xff9b) // NAT64 (could reach IPv4 private)
|| s[0] == 0x2002 // 6to4 (embeds any IPv4)
|| (s[0] == 0x2001 && s[1] == 0)) // Teredo
}
}
}
// ---- egress proxy ---------------------------------------------------------------------------------
/// A loopback HTTP proxy that connects only to addresses `allow` accepts (`is_public` for Chrome).
pub struct EgressProxy {
pub port: u16,
task: tokio::task::JoinHandle<()>,
}
impl EgressProxy {
pub async fn start(allow: fn(&IpAddr) -> bool) -> std::io::Result<Self> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let port = listener.local_addr()?.port();
let task = tokio::spawn(async move {
while let Ok((client, _)) = listener.accept().await {
tokio::spawn(async move {
let _ = serve(client, allow).await;
});
}
});
Ok(Self { port, task })
}
pub fn stop(&self) {
self.task.abort();
}
}
impl Drop for EgressProxy {
fn drop(&mut self) {
self.task.abort();
}
}
/// The request head (up to the blank line) and whatever came after it.
async fn read_head(s: &mut TcpStream) -> std::io::Result<Option<(String, Vec<u8>)>> {
let mut buf = Vec::new();
let mut chunk = [0u8; 8192];
loop {
if let Some(end) = buf.windows(4).position(|w| w == b"\r\n\r\n") {
let rest = buf.split_off(end + 4);
return Ok(String::from_utf8(buf).ok().map(|h| (h, rest)));
}
if buf.len() > MAX_HEAD {
return Ok(None);
}
let n = s.read(&mut chunk).await?;
if n == 0 {
return Ok(None);
}
buf.extend_from_slice(&chunk[..n]);
}
}
/// `host:port` with an optional `[..]` around an IPv6 address.
fn split_host_port(s: &str) -> Option<(String, u16)> {
let (host, port) = s.rsplit_once(':')?;
let host = host.strip_prefix('[').and_then(|h| h.strip_suffix(']')).unwrap_or(host);
if host.is_empty() {
return None;
}
Some((host.to_owned(), port.parse().ok()?))
}
/// Resolve once, require every address to be allowed, and connect only to those.
async fn connect_allowed(host: &str, port: u16, allow: fn(&IpAddr) -> bool) -> Result<TcpStream, &'static str> {
let addrs: Vec<SocketAddr> = match host.parse::<IpAddr>() {
Ok(ip) => vec![SocketAddr::new(ip, port)],
Err(_) => match tokio::time::timeout(RESOLVE_TIMEOUT, tokio::net::lookup_host((host, port))).await {
Ok(Ok(a)) => a.collect(),
_ => return Err("name not resolved"),
},
};
if addrs.is_empty() || !addrs.iter().all(|a| allow(&a.ip())) {
return Err("not a public address");
}
for a in &addrs {
if let Ok(Ok(s)) = tokio::time::timeout(CONNECT_TIMEOUT, TcpStream::connect(a)).await {
return Ok(s);
}
}
Err("could not connect")
}
async fn reply(client: &mut TcpStream, status: &str) -> std::io::Result<()> {
client.write_all(format!("HTTP/1.1 {status}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n").as_bytes()).await
}
async fn serve(mut client: TcpStream, allow: fn(&IpAddr) -> bool) -> std::io::Result<()> {
let Ok(Ok(Some((head, rest)))) = tokio::time::timeout(HEAD_TIMEOUT, read_head(&mut client)).await else {
return reply(&mut client, "400 Bad Request").await;
};
let mut lines = head.split("\r\n");
let mut first = lines.next().unwrap_or("").split(' ');
let (method, target, version) = (first.next().unwrap_or(""), first.next().unwrap_or(""), first.next().unwrap_or(""));
if method.eq_ignore_ascii_case("CONNECT") {
let Some((host, port)) = split_host_port(target) else { return reply(&mut client, "400 Bad Request").await };
let mut up = match connect_allowed(&host, port, allow).await {
Ok(s) => s,
Err(why) => return refused(&mut client, why).await,
};
client.write_all(b"HTTP/1.1 200 Connection Established\r\n\r\n").await?;
up.write_all(&rest).await?;
tokio::io::copy_bidirectional(&mut client, &mut up).await?;
return Ok(());
}
let Ok(url) = reqwest::Url::parse(target) else { return reply(&mut client, "400 Bad Request").await };
let (Some(host), Some(port)) = (url.host_str(), url.port_or_known_default()) else {
return reply(&mut client, "400 Bad Request").await;
};
if url.scheme() != "http" || !version.starts_with("HTTP/") {
return reply(&mut client, "400 Bad Request").await;
}
let host = host.trim_start_matches('[').trim_end_matches(']').to_owned();
let mut up = match connect_allowed(&host, port, allow).await {
Ok(s) => s,
Err(why) => return refused(&mut client, why).await,
};
// Origin form, no proxy headers, one request per connection (the next may be for another host).
let path = match url.query() {
Some(q) => format!("{}?{q}", url.path()),
None => url.path().to_owned(),
};
let mut out = format!("{method} {path} {version}\r\n");
for l in lines.filter(|l| !l.is_empty()) {
let name = l.split(':').next().unwrap_or("").trim().to_ascii_lowercase();
if !matches!(name.as_str(), "proxy-connection" | "proxy-authorization" | "connection" | "keep-alive") {
out.push_str(l);
out.push_str("\r\n");
}
}
out.push_str("Connection: close\r\n\r\n");
up.write_all(out.as_bytes()).await?;
up.write_all(&rest).await?;
tokio::io::copy_bidirectional(&mut client, &mut up).await?;
Ok(())
}
async fn refused(client: &mut TcpStream, why: &str) -> std::io::Result<()> {
// Never the host or the address: they are what the bot was asked to open.
crate::debug_log!("browser", "proxy refused a connection: {why}");
reply(client, "403 Forbidden").await
}
// ---- Chrome ---------------------------------------------------------------------------------------
/// The arguments the bot's Chrome runs with (headless, its profile, CDP on a free port, every
/// connection through the proxy on `proxy_port`).
pub fn chrome_args(profile: &Path, proxy_port: u16, no_sandbox: bool) -> Vec<String> {
let mut a = vec![
"--headless=new".to_owned(),
format!("--user-data-dir={}", profile.display()),
"--remote-debugging-port=0".into(),
format!("--proxy-server=http://127.0.0.1:{proxy_port}"),
"--proxy-bypass-list=<-loopback>".into(),
"--disable-quic".into(),
"--force-webrtc-ip-handling-policy=disable_non_proxied_udp".into(),
"--no-first-run".into(),
"--no-default-browser-check".into(),
"--disable-sync".into(),
"--disable-background-networking".into(),
"--disable-component-update".into(),
"--disable-default-apps".into(),
"--disable-features=Translate,MediaRouter,OptimizationHints".into(),
"--window-size=1280,800".into(),
];
if no_sandbox {
a.push("--no-sandbox".into());
}
a.push("about:blank".into());
a
}
pub struct BotBrowser {
child: Mutex<Option<tokio::process::Child>>,
proxy: EgressProxy,
pub cdp_port: u16,
}
impl BotBrowser {
/// Start Chrome `exe` with profile folder `profile` and wait until CDP answers.
pub async fn start(exe: &Path, profile: &Path) -> Result<Self, String> {
let proxy = EgressProxy::start(is_public).await.map_err(|e| format!("proxy: {e}"))?;
std::fs::create_dir_all(profile).map_err(|e| e.to_string())?;
let port_file = profile.join("DevToolsActivePort");
let _ = std::fs::remove_file(&port_file);
let no_sandbox = cfg!(not(windows)) && std::env::var("JARVIS_CHROME_NO_SANDBOX").is_ok_and(|v| v == "1");
let mut cmd = tokio::process::Command::new(exe);
cmd.args(chrome_args(profile, proxy.port, no_sandbox)).stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::null()).stderr(std::process::Stdio::null()).kill_on_drop(true);
#[cfg(unix)]
cmd.process_group(0);
#[cfg(windows)]
cmd.creation_flags(0x0800_0000); // CREATE_NO_WINDOW
let mut child = cmd.spawn().map_err(|e| format!("Chrome을 실행하지 못했습니다: {e}"))?;
let until = tokio::time::Instant::now() + READY_TIMEOUT;
let cdp_port = loop {
if let Some(p) = std::fs::read_to_string(&port_file).ok()
.and_then(|t| t.lines().next().and_then(|l| l.trim().parse::<u16>().ok())) {
if version(p).await.is_some() {
break p;
}
}
if let Ok(Some(status)) = child.try_wait() {
return Err(format!("Chrome이 바로 끝났습니다 ({status})"));
}
if tokio::time::Instant::now() > until {
kill_tree(&mut child).await;
return Err("Chrome이 30초 안에 준비되지 않았습니다".into());
}
tokio::time::sleep(Duration::from_millis(200)).await;
};
crate::info_log!("browser", "bot browser ready (CDP on 127.0.0.1:{cdp_port})");
Ok(Self { child: Mutex::new(Some(child)), proxy, cdp_port })
}
pub fn cdp_url(&self) -> String {
format!("http://127.0.0.1:{}", self.cdp_port)
}
/// Whether Chrome still runs.
pub fn running(&self) -> bool {
self.child.lock().unwrap().as_mut().is_some_and(|c| matches!(c.try_wait(), Ok(None)))
}
/// `Browser.close`, then up to 5 s for Chrome to end, else it is killed; the proxy ends too.
pub async fn stop(&self) {
let child = self.child.lock().unwrap().take();
if let Some(mut child) = child {
if let Some(ws) = version(self.cdp_port).await.and_then(|v| v["webSocketDebuggerUrl"].as_str().map(str::to_owned)) {
let _ = close_over_cdp(&ws).await;
}
if !matches!(tokio::time::timeout(CLOSE_WAIT, child.wait()).await, Ok(Ok(_))) {
crate::warn_log!("browser", "Chrome did not close in {} s; killing it", CLOSE_WAIT.as_secs());
kill_tree(&mut child).await;
}
crate::info_log!("browser", "bot browser stopped");
}
self.proxy.stop();
}
}
/// CDP's `/json/version`, if it answers.
async fn version(port: u16) -> Option<serde_json::Value> {
let client = reqwest::Client::builder().no_proxy().timeout(Duration::from_secs(2)).build().ok()?;
let text = client.get(format!("http://127.0.0.1:{port}/json/version")).send().await.ok()?.text().await.ok()?;
serde_json::from_str(&text).ok()
}
async fn close_over_cdp(ws: &str) -> Result<(), String> {
use futures_util::SinkExt;
let (mut s, _) = tokio::time::timeout(Duration::from_secs(2), tokio_tungstenite::connect_async(ws)).await
.map_err(|_| "timeout".to_string())?.map_err(|e| e.to_string())?;
s.send(tokio_tungstenite::tungstenite::Message::text(r#"{"id":1,"method":"Browser.close"}"#)).await
.map_err(|e| e.to_string())
}
async fn kill_tree(child: &mut tokio::process::Child) {
if let Some(pid) = child.id() {
#[cfg(unix)]
// SAFETY: a plain signal to Chrome's own process group (made with process_group(0)).
unsafe {
libc::kill(-(pid as i32), libc::SIGKILL);
}
#[cfg(windows)]
{
let mut tk = tokio::process::Command::new("taskkill");
tk.args(["/T", "/F", "/PID", &pid.to_string()]).stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::null()).stderr(std::process::Stdio::null()).creation_flags(0x0800_0000);
let _ = tk.status().await;
}
}
let _ = child.kill().await;
}
/// Where the bot browser's profile lives in the data folder.
pub fn profile_dir(data: &Path) -> PathBuf {
data.join("browser").join("profile")
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[test]
fn only_public_addresses_are_public() {
let public = ["8.8.8.8", "1.1.1.1", "142.250.1.1", "2606:4700:4700::1111", "2a00:1450:4001::1"];
let not = ["127.0.0.1", "10.0.0.1", "172.16.5.4", "192.168.10.9", "169.254.169.254", "100.64.0.1",
"0.0.0.0", "0.1.2.3", "192.0.0.8", "192.0.2.1", "198.18.0.1", "203.0.113.5", "224.0.0.1", "240.0.0.1",
"255.255.255.255", "::1", "::", "fc00::1", "fd12::1", "fe80::1", "ff02::1", "2001:db8::1",
"::ffff:127.0.0.1", "::ffff:192.168.0.1", "64:ff9b::a00:1", "2002:c0a8:1::1", "2001:0:4136:e378::1"];
for a in public {
assert!(is_public(&a.parse().unwrap()), "{a} should be public");
}
for a in not {
assert!(!is_public(&a.parse().unwrap()), "{a} should not be public");
}
}
/// A local server recording each connection's first bytes and answering `answer`.
async fn server(answer: &'static [u8]) -> (u16, Arc<Mutex<Vec<String>>>) {
let l = TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = l.local_addr().unwrap().port();
let seen = Arc::new(Mutex::new(Vec::new()));
let s2 = seen.clone();
tokio::spawn(async move {
while let Ok((mut c, _)) = l.accept().await {
let seen = s2.clone();
tokio::spawn(async move {
let mut buf = vec![0u8; 4096];
let n = c.read(&mut buf).await.unwrap_or(0);
seen.lock().unwrap().push(String::from_utf8_lossy(&buf[..n]).into_owned());
let _ = c.write_all(answer).await;
});
}
});
(port, seen)
}
async fn ask(proxy: u16, req: String) -> String {
let mut c = TcpStream::connect(("127.0.0.1", proxy)).await.unwrap();
c.write_all(req.as_bytes()).await.unwrap();
let mut out = Vec::new();
let _ = tokio::time::timeout(Duration::from_secs(5), c.read_to_end(&mut out)).await;
String::from_utf8_lossy(&out).into_owned()
}
fn loopback(ip: &IpAddr) -> bool {
ip.is_loopback()
}
#[tokio::test]
async fn the_proxy_refuses_this_pc_and_the_local_network() {
let (port, seen) = server(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nhi").await;
let p = EgressProxy::start(is_public).await.unwrap();
for req in [format!("CONNECT 127.0.0.1:{port} HTTP/1.1\r\nHost: x\r\n\r\n"),
format!("CONNECT localhost:{port} HTTP/1.1\r\n\r\n"),
format!("CONNECT [::1]:{port} HTTP/1.1\r\n\r\n"),
format!("GET http://127.0.0.1:{port}/a HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"),
format!("GET http://localhost:{port}/a HTTP/1.1\r\n\r\n"),
"CONNECT 192.168.10.1:443 HTTP/1.1\r\n\r\n".to_string(),
"GET http://10.0.0.1/ HTTP/1.1\r\n\r\n".to_string()] {
let r = ask(p.port, req.clone()).await;
assert!(r.starts_with("HTTP/1.1 403"), "{req:?} -> {r:?}");
}
assert!(seen.lock().unwrap().is_empty());
for bad in ["nonsense\r\n\r\n", "GET /relative HTTP/1.1\r\n\r\n", "GET ftp://a.example/ HTTP/1.1\r\n\r\n",
"CONNECT nohost HTTP/1.1\r\n\r\n"] {
assert!(ask(p.port, bad.into()).await.starts_with("HTTP/1.1 400"), "{bad:?}");
}
}
#[tokio::test]
async fn the_proxy_tunnels_and_forwards_to_allowed_addresses() {
let (port, seen) = server(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nhi").await;
let p = EgressProxy::start(loopback).await.unwrap();
let r = ask(p.port, format!("CONNECT 127.0.0.1:{port} HTTP/1.1\r\n\r\nping")).await;
assert!(r.starts_with("HTTP/1.1 200 Connection Established\r\n\r\nHTTP/1.1 200 OK"), "{r:?}");
let r = ask(p.port, format!("GET http://127.0.0.1:{port}/a/b?c=1 HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n\
Proxy-Connection: keep-alive\r\nConnection: keep-alive\r\nAccept: */*\r\n\r\n")).await;
assert!(r.ends_with("hi"), "{r:?}");
let seen = seen.lock().unwrap().clone();
assert_eq!(seen[0], "ping");
assert_eq!(seen[1], format!("GET /a/b?c=1 HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\nAccept: */*\r\nConnection: close\r\n\r\n"));
}
#[test]
fn chrome_goes_through_the_proxy_with_its_own_profile() {
let a = chrome_args(Path::new("/d/browser/profile"), 4321, false);
for want in ["--headless=new", "--user-data-dir=/d/browser/profile", "--remote-debugging-port=0",
"--proxy-server=http://127.0.0.1:4321", "--proxy-bypass-list=<-loopback>", "--disable-quic"] {
assert!(a.iter().any(|x| x == want), "{want}");
}
assert!(!a.iter().any(|x| x == "--no-sandbox"));
assert!(chrome_args(Path::new("/p"), 1, true).iter().any(|x| x == "--no-sandbox"));
}
}

View File

@@ -269,6 +269,7 @@ impl Controller {
view["missed"] = status["missed"].clone();
view["worker"] = status["worker"].clone();
view["voiceMembers"] = status["voiceMembers"].clone();
view["browser"] = status.get("browser").cloned().unwrap_or_else(|| json!({"state": "off", "note": null}));
view["state"] = json!(state);
view["error"] = json!(error);
view["system"] = json!(self.system().await);
@@ -360,6 +361,7 @@ impl Running for crate::services::Services {
"voiceMembers": self.core.voice_members().into_iter()
.map(|(id, name)| json!({"id": id.to_string(), "name": name})).collect::<Vec<_>>(),
"missed": crate::dashboard::missed_view(&missed, now),
"browser": self.browser.view(),
})
}
}

View File

@@ -100,9 +100,9 @@ mod tests {
#[test]
fn features_that_are_off_turn_their_tools_off() {
let all_on = FeatureSettings { weather: true, web_search: true, meals: true, collect: true };
let all_on = FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true };
assert!(disabled_tools(&all_on).is_empty());
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false, collect: false });
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false, collect: false, browser: false });
assert_eq!(off, vec!["getWeather", "logMeal", "fetchMeals", "deleteMeal"]);
}

View File

@@ -32,6 +32,15 @@ const TORCH: [&str; 2] = ["torch==2.11.0", "torchaudio==2.11.0"];
const TORCH_PIN: &str = "torch==2.11.0+cu128\ntorchaudio==2.11.0+cu128\n";
const MELO_URL: &str = "https://github.com/myshell-ai/MeloTTS/archive/209145371cff8fc3bd60d7be902ea69cbdb7965a.tar.gz";
pub const STT_MODEL: &str = "large-v3-turbo";
/// The bot browser (admin.spec.md, "Bot browser"): Chrome for Testing, per platform the archive
/// and its SHA-256 (computed from the downloads; Google publishes none).
pub const CHROME_VERSION: &str = "157.0.8103.78";
#[cfg(windows)]
const CHROME_ARCHIVE: (&str, &str) = ("win64/chrome-win64.zip",
"5f6ec40206407112702e163bf694d07f9fdc8ccbd346c70ce61fa62f716493f9");
#[cfg(not(windows))]
const CHROME_ARCHIVE: (&str, &str) = ("linux64/chrome-linux64.zip",
"9223832977a13db3646201b1341c1f9cf08afa3ceae92540116d7b0c062364fb");
/// Where everything goes, from the data folder and the app's Python sources (`voice_worker`, `jarvis`).
#[derive(Debug, Clone)]
@@ -61,6 +70,10 @@ impl Layout {
self.strata_home().join("terminal").join("current").join("bin")
.join(if cfg!(windows) { "strata.cmd" } else { "strata" })
}
pub fn chrome_exe(&self) -> PathBuf {
let dir = self.runtime().join("chrome");
if cfg!(windows) { dir.join("chrome-win64").join("chrome.exe") } else { dir.join("chrome-linux64").join("chrome") }
}
/// Hugging Face models (Whisper, MeloTTS and its BERT); the worker gets the same `HF_HOME`.
pub fn hf_home(&self) -> PathBuf {
self.data.join("models").join("hf")
@@ -346,6 +359,11 @@ impl Installer {
serde_json::json!({"running": self.running.load(Ordering::SeqCst), "done": all_done, "steps": steps})
}
/// Whether the step `id` is installed at its current version.
pub fn done(&self, id: &str) -> bool {
self.steps.iter().any(|s| s.id() == id && self.is_done(s.as_ref()))
}
pub fn all_done(&self) -> bool {
self.steps.iter().all(|s| self.is_done(s.as_ref()))
}
@@ -600,6 +618,55 @@ impl Step for StrataStep {
}
}
struct BrowserStep;
#[async_trait]
impl Step for BrowserStep {
fn id(&self) -> &'static str { "browser" }
fn label(&self) -> &'static str { "봇용 크롬" }
fn size_hint(&self) -> &'static str { "약 200MB" }
fn version(&self, _: &Layout) -> String { CHROME_VERSION.into() }
async fn run(&self, ctx: &Ctx) -> Result<(), String> {
let (path, sum) = CHROME_ARCHIVE;
let url = format!("https://storage.googleapis.com/chrome-for-testing-public/{CHROME_VERSION}/{path}");
let archive = ctx.layout.data.join("cache").join("downloads").join(path.rsplit('/').next().unwrap());
crate::download::download(&url, &archive, sum, &|d, t| ctx.bytes(d, t), &ctx.cancel).await.map_err(|e| e.0)?;
let dir = ctx.layout.runtime().join("chrome");
let _ = std::fs::remove_dir_all(&dir);
unzip(&archive, &dir)?;
let _ = std::fs::remove_file(&archive);
if !ctx.layout.chrome_exe().exists() {
return Err("크롬 압축을 풀었지만 실행 파일이 없습니다".into());
}
Ok(())
}
}
/// Unpack a zip into `dir`, keeping its folders and (on Unix) its executable bits.
fn unzip(archive: &Path, dir: &Path) -> Result<(), String> {
let f = std::fs::File::open(archive).map_err(|e| e.to_string())?;
let mut z = zip::ZipArchive::new(f).map_err(|e| e.to_string())?;
for i in 0..z.len() {
let mut entry = z.by_index(i).map_err(|e| e.to_string())?;
let Some(rel) = entry.enclosed_name() else { continue };
let out = dir.join(rel);
if entry.is_dir() {
std::fs::create_dir_all(&out).map_err(|e| e.to_string())?;
continue;
}
if let Some(parent) = out.parent() {
std::fs::create_dir_all(parent).map_err(|e| e.to_string())?;
}
let mut w = std::fs::File::create(&out).map_err(|e| e.to_string())?;
std::io::copy(&mut entry, &mut w).map_err(|e| e.to_string())?;
#[cfg(unix)]
if let Some(mode) = entry.unix_mode() {
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(&out, std::fs::Permissions::from_mode(mode & 0o777)).map_err(|e| e.to_string())?;
}
}
Ok(())
}
/// Point the services at what was installed: the worker venv's Python and the arguments for the
/// MeloTTS venv, the history store and the brain (`voiceWorker` in `config/settings.json`; other
/// keys are kept).
@@ -638,7 +705,7 @@ impl Step for SettingsStep {
/// The install steps, in order.
pub fn steps() -> Vec<Box<dyn Step>> {
vec![Box::new(UvStep), Box::new(WorkerEnvStep), Box::new(MeloEnvStep), Box::new(SttModelStep),
Box::new(TtsModelStep), Box::new(StrataStep), Box::new(SettingsStep)]
Box::new(TtsModelStep), Box::new(StrataStep), Box::new(BrowserStep), Box::new(SettingsStep)]
}
#[cfg(test)]
@@ -823,4 +890,30 @@ mod tests {
}
assert!(l.uv().starts_with(&l.data) && l.venv_python("worker").starts_with(&l.data));
}
#[test]
fn a_zip_unpacks_with_its_folders_and_executable_bits() {
use std::io::Write;
let d = temp_dir("unzip");
let archive = d.join("a.zip");
let mut w = zip::ZipWriter::new(std::fs::File::create(&archive).unwrap());
let exe = zip::write::SimpleFileOptions::default().unix_permissions(0o755);
w.add_directory("top/sub/", zip::write::SimpleFileOptions::default()).unwrap();
w.start_file("top/chrome", exe).unwrap();
w.write_all(b"#!/bin/sh\n").unwrap();
w.start_file("top/sub/data.pak", zip::write::SimpleFileOptions::default().unix_permissions(0o644)).unwrap();
w.write_all(b"data").unwrap();
w.start_file("../escape", zip::write::SimpleFileOptions::default()).unwrap();
w.write_all(b"no").unwrap();
w.finish().unwrap();
let out = d.join("out");
unzip(&archive, &out).unwrap();
assert_eq!(std::fs::read(out.join("top/sub/data.pak")).unwrap(), b"data");
assert!(!d.join("escape").exists());
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
assert_eq!(std::fs::metadata(out.join("top/chrome")).unwrap().permissions().mode() & 0o777, 0o755);
}
}
}

View File

@@ -2,6 +2,7 @@
pub mod log;
pub mod admin_api;
pub mod browser;
pub mod calls;
pub mod channel_sync;
pub mod chat;

View File

@@ -22,6 +22,34 @@ pub struct Services {
/// The conversation's missed requests, for the dashboard.
pub missed: conversation::MissedLog,
pub llm: Arc<Llm>,
pub browser: BrowserRun,
}
/// The bot browser of a run (admin.spec.md, "Bot browser").
pub enum BrowserRun {
/// Not started: the feature is off, or (with a note) it is not installed.
Off(Option<String>),
Failed(String),
Running(Arc<crate::browser::BotBrowser>),
}
impl BrowserRun {
/// `{state, note}` for the dashboard.
pub fn view(&self) -> Value {
let (state, note) = match self {
BrowserRun::Off(note) => ("off", note.clone()),
BrowserRun::Failed(e) => ("failed", Some(e.clone())),
BrowserRun::Running(b) if b.running() => ("ready", None),
BrowserRun::Running(_) => ("exited", None),
};
serde_json::json!({"state": state, "note": note})
}
pub async fn stop(&self) {
if let BrowserRun::Running(b) = self {
b.stop().await;
}
}
}
impl Services {
@@ -30,10 +58,31 @@ impl Services {
self.core.close().await;
self.worker.close();
self.worker_process.stop().await;
self.browser.stop().await;
self.llm.stop().await
}
}
/// Start the bot browser when its feature is on and it is installed; a failure leaves the services
/// running without it.
async fn start_browser(data_dir: &Path, settings: &crate::settings::Settings) -> BrowserRun {
if !settings.features.browser {
return BrowserRun::Off(None);
}
let layout = crate::install::Layout { data: data_dir.to_path_buf(), src: std::path::PathBuf::new() };
if !crate::install::Installer::new(layout.clone(), crate::install::steps()).done("browser") {
crate::warn_log!("browser", "the bot browser is not installed");
return BrowserRun::Off(Some("설치되지 않음".into()));
}
match crate::browser::BotBrowser::start(&layout.chrome_exe(), &crate::browser::profile_dir(data_dir)).await {
Ok(b) => BrowserRun::Running(Arc::new(b)),
Err(e) => {
crate::warn_log!("browser", "Warning: the bot browser did not start: {e}");
BrowserRun::Failed(e)
}
}
}
/// Start the services; the LLM first (its first start downloads the model), so nothing answers
/// before the brain can. A failure stops what already started.
pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services, Box<dyn std::error::Error + Send + Sync>> {
@@ -60,24 +109,28 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
Ok(f) => f,
Err(e) => return Err(after_llm_stop(&llm, format!("could not write the brain settings: {e}")).await.into()),
};
let browser = start_browser(data_dir, &settings).await;
let w = &settings.voice_worker;
let mut wopts = WorkerProcessOptions::new(&w.python, &w.cwd, settings.worker_args());
// With the installed runtime, the worker's models, caches and temp files stay in the data folder.
let layout = crate::install::Layout { data: data_dir.to_path_buf(), src: w.cwd.clone() };
if crate::install::Installer::new(layout.clone(), crate::install::steps()).all_done() {
if crate::install::Installer::new(layout.clone(), crate::install::steps()).done("settings") {
wopts.env.extend(layout.env());
}
wopts.env.insert("JARVIS_CONFIG_PATH".into(), brain_config.to_string_lossy().into_owned());
wopts.env.insert(crate::features::MCP_ENV_VAR.into(), mcp_env.to_string());
if let BrowserRun::Running(b) = &browser {
wopts.env.insert("JARVIS_BROWSER_CDP".into(), b.cdp_url());
}
// Background collection (admin.spec.md): the enabled targets, and the keep days in any case.
match crate::collect::write_config(data_dir, &settings) {
Ok(f) => { wopts.env.insert(crate::collect::COLLECT_ENV_VAR.into(), f.to_string_lossy().into_owned()); }
Err(e) => return Err(after_llm_stop(&llm, format!("could not write the collection settings: {e}")).await.into()),
Err(e) => return Err(after_stop(&llm, &browser, format!("could not write the collection settings: {e}")).await.into()),
}
let worker_process = match WorkerProcess::new(wopts) {
Ok(p) => p,
Err(e) => return Err(after_llm_stop(&llm, e.to_string()).await.into()),
Err(e) => return Err(after_stop(&llm, &browser, e.to_string()).await.into()),
};
let mut process_events = worker_process.subscribe();
worker_process.start();
@@ -111,10 +164,10 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
if let Err(e) = core.start().await {
worker.close();
worker_process.stop().await;
return Err(after_llm_stop(&llm, e.to_string()).await.into());
return Err(after_stop(&llm, &browser, e.to_string()).await.into());
}
crate::info_log!("app", "services started");
Ok(Services { core, events, worker, worker_process, missed, llm })
Ok(Services { core, events, worker, worker_process, missed, llm, browser })
}
/// The secret values the services get: the bot token and the MCP servers' environment values.
@@ -126,6 +179,12 @@ pub fn secrets_in_use(token: &str, mcp_env: &Value) -> Vec<String> {
v
}
/// Stop the bot browser and the LLM after a failed start.
async fn after_stop(llm: &Llm, browser: &BrowserRun, error: String) -> String {
browser.stop().await;
after_llm_stop(llm, error).await
}
/// Stop the LLM after a failed start; the error also says if its server may still run.
async fn after_llm_stop(llm: &Llm, error: String) -> String {
match llm.stop().await {

View File

@@ -93,6 +93,8 @@ pub struct FeatureSettings {
pub meals: bool,
/// Background collection (admin.spec.md, "Background collection").
pub collect: bool,
/// The bot browser (admin.spec.md, "Bot browser").
pub browser: bool,
}
/// Background collection (admin.spec.md, "Background collection"); no targets by default.
@@ -184,7 +186,7 @@ impl Default for Settings {
response: PolicySettings::default(),
memory: Map::new(),
llm: LlmSettings { url: None, port: 8080, context: 32768 },
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true },
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true },
mcp_servers: vec![],
collect: CollectSettings { keep_days: 30, targets: vec![] },
}
@@ -505,6 +507,7 @@ pub fn parse_settings(raw: &Value) -> Result<Settings, SettingsError> {
web_search: get_bool(f, "webSearch", "features", d.features.web_search)?,
meals: get_bool(f, "meals", "features", d.features.meals)?,
collect: get_bool(f, "collect", "features", d.features.collect)?,
browser: get_bool(f, "browser", "features", d.features.browser)?,
}
},
mcp_servers: parse_mcp_servers(raw)?,
@@ -804,13 +807,13 @@ mod tests {
#[test]
fn features_and_mcp_servers_are_read_and_checked() {
let s = parse_settings(&json!({}));
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true, collect: true });
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true });
let s = parse_settings(&json!({
"features": {"weather": false, "meals": false},
"mcpServers": [{"name": "brave", "command": "npx", "args": ["-y", "brave"], "env": ["BRAVE_API_KEY", " "]},
{"name": "time-2", "command": "uvx", "enabled": false}]
})).unwrap();
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false, collect: true });
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false, collect: true, browser: true });
assert_eq!(s.mcp_servers, vec![
McpServerSettings { name: "brave".into(), command: "npx".into(), args: vec!["-y".into(), "brave".into()],
env: vec!["BRAVE_API_KEY".into()], enabled: true },

View File

@@ -123,6 +123,7 @@ pub fn fields() -> Vec<Field> {
f("features.webSearch", Features, Bool, json!(true), "웹 검색"),
f("features.meals", Features, Bool, json!(true), "식단 기록"),
f("features.collect", Features, Bool, json!(true), "백그라운드 수집"),
f("features.browser", Features, Bool, json!(true), "브라우저 제어(봇용 크롬)"),
f("mcpServers", Features, McpServers, json!([]), "MCP 서버"),
// Background collection (admin.spec.md, "Background collection"): no targets by default.
f("collect.targets", Collect, CollectTargets, json!([]), "수집 대상"),

87
app/tests/bot_browser.rs Normal file
View File

@@ -0,0 +1,87 @@
//! The bot browser with a real Chrome (admin.spec.md, "Bot browser"). Runs only with
//! `JARVIS_TEST_CHROME=<chrome executable>`; needs the internet for the public page. On Ubuntu also
//! set `JARVIS_CHROME_NO_SANDBOX=1`.
use std::sync::{Arc, Mutex};
use std::time::Duration;
use jarvis_app::browser::BotBrowser;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
async fn titles(client: &reqwest::Client, port: u16) -> Vec<(String, String)> {
let text = client.get(format!("http://127.0.0.1:{port}/json/list")).send().await.unwrap().text().await.unwrap();
let list: serde_json::Value = serde_json::from_str(&text).unwrap();
list.as_array().unwrap().iter()
.map(|t| (t["url"].as_str().unwrap_or("").to_owned(), t["title"].as_str().unwrap_or("").to_owned())).collect()
}
#[tokio::test]
async fn the_bot_browser_reaches_public_pages_never_this_pc_and_stops() {
let Ok(exe) = std::env::var("JARVIS_TEST_CHROME") else {
eprintln!("skipped: set JARVIS_TEST_CHROME");
return;
};
// A page on this PC the browser must never reach.
let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let local = l.local_addr().unwrap().port();
let hits = Arc::new(Mutex::new(0));
let h2 = hits.clone();
tokio::spawn(async move {
while let Ok((mut c, _)) = l.accept().await {
*h2.lock().unwrap() += 1;
let mut b = [0u8; 1024];
let _ = c.read(&mut b).await;
let _ = c.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 30\r\n\r\n<title>secret local</title>xx").await;
}
});
let profile = std::env::temp_dir().join(format!("jarvis-bot-browser-{}", std::process::id()));
let b = BotBrowser::start(std::path::Path::new(&exe), &profile).await.expect("chrome starts");
assert!(b.running());
let client = reqwest::Client::builder().no_proxy().build().unwrap();
for url in [format!("http://127.0.0.1:{local}/"), format!("http://localhost:{local}/"), "https://example.com/".into()] {
client.put(format!("http://127.0.0.1:{}/json/new?{url}", b.cdp_port)).send().await.unwrap();
}
let mut got = vec![];
for _ in 0..40 {
got = titles(&client, b.cdp_port).await;
if got.iter().any(|(_, t)| t == "Example Domain") {
break;
}
tokio::time::sleep(Duration::from_millis(250)).await;
}
eprintln!("tabs: {got:?}");
assert!(got.iter().any(|(_, t)| t == "Example Domain"), "the public page loads through the proxy");
assert!(!got.iter().any(|(_, t)| t.contains("secret local")));
assert_eq!(*hits.lock().unwrap(), 0, "nothing reached this PC");
let port = b.cdp_port;
let t0 = std::time::Instant::now();
b.stop().await;
eprintln!("stopped in {:?}", t0.elapsed());
assert!(!b.running());
assert!(client.get(format!("http://127.0.0.1:{port}/json/version")).send().await.is_err(), "CDP is closed");
let _ = std::fs::remove_dir_all(&profile);
}
/// The install step downloads the pinned Chrome (about 200 MB) and unpacks a runnable browser.
/// Runs only with `JARVIS_TEST_CHROME_INSTALL=1`.
#[tokio::test]
async fn the_browser_step_installs_a_chrome_that_starts() {
if std::env::var("JARVIS_TEST_CHROME_INSTALL").as_deref() != Ok("1") {
eprintln!("skipped: set JARVIS_TEST_CHROME_INSTALL=1");
return;
}
let data = std::env::temp_dir().join(format!("jarvis-chrome-install-{}", std::process::id()));
let layout = jarvis_app::install::Layout { data: data.clone(), src: std::path::PathBuf::new() };
let steps = jarvis_app::install::steps().into_iter().filter(|s| s.id() == "browser").collect();
let installer = jarvis_app::install::Installer::new(layout.clone(), steps);
assert!(installer.run_all().await, "{}", installer.status());
assert!(installer.done("browser"));
assert!(!data.join("cache").join("downloads").read_dir().map(|mut d| d.next().is_some()).unwrap_or(false),
"the archive is removed after unpacking");
let b = BotBrowser::start(&layout.chrome_exe(), &jarvis_app::browser::profile_dir(&data)).await.expect("starts");
assert!(b.running());
b.stop().await;
let _ = std::fs::remove_dir_all(&data);
}

View File

@@ -42,6 +42,7 @@ const CHOICE = {
};
const REASON = { busy: "다른 답을 하는 중", queue_full: "대기열이 가득 참", replaced: "더 새 요청이 대신함",
interrupted: "끼어들어 멈춤", lower_score: "다른 요청이 더 중요함", stopped: "정지 명령" };
const BROWSER = { off: "꺼짐", ready: "준비", failed: "실패", exited: "꺼짐(스스로 끝남)" };
const STATE = { stopped: ["꺼짐", ""], starting: ["켜는 중", "warn"], running: ["실행 중", "ok"],
stopping: ["끄는 중", "warn"], failed: ["실패", "bad"] };
@@ -426,7 +427,9 @@ function dashboardBody(d) {
h("span", { class: "k" }, "음성 워커"), h("span", {}, w ? (w.connected ? "연결됨" : "연결 안 됨") : "–"),
h("span", { class: "k" }, "STT / TTS"), h("span", {}, w ? `${w.sttReady ? "준비" : "–"} / ${w.ttsReady ? "준비" : "–"}` : "–"),
h("span", { class: "k" }, "워커 재시작"), h("span", {}, w?.restarts ?? "–"),
h("span", { class: "k" }, "대화 모델"), h("span", {}, llm ? (LLM[llm.state] ?? llm.state) : "–"))),
h("span", { class: "k" }, "대화 모델"), h("span", {}, llm ? (LLM[llm.state] ?? llm.state) : "–"),
h("span", { class: "k" }, "봇용 크롬"), h("span", { title: d.browser?.note ?? "" },
(BROWSER[d.browser?.state] ?? "–") + (d.browser?.note ? ` (${d.browser.note})` : "")))),
h("div", { class: "card" }, h("h2", {}, "음성 채널"),
h("div", {}, d.voiceChannel ? `채널 ${d.voiceChannel}` : "들어가 있지 않음"),
h("div", { class: "small muted", style: "margin-top:6px" },

File diff suppressed because one or more lines are too long