feat(app): the bot browser's window on a virtual monitor to share in Discord

- features.browserShare (봇용 크롬창 화면공유 선택 표시, off by default) and a driver card on the
  features screen: the Virtual Display Driver 25.7.23 (MIT, SignPath-signed) ships in
  display-driver/ (hash-checked by package.sh); install and uninstall run jarvis.exe elevated
  once (--display-driver), settings in %ProgramData%\JarvisBot\display-driver, another
  program's copy of the driver is left alone
- the monitor is on the desktop only while the bot browser uses it, at the rightmost real
  monitor's bottom-right corner; the stop closes Chrome first, then takes the monitor off
- Chrome starts with no window and the app opens it over CDP in the background, so it never
  takes the focus; the window is a tool window (no Alt+Tab, no taskbar button), fills the
  monitor's work area and keeps drawing when covered; without a monitor it runs headless with
  the reason on the dashboard
- Chrome runs with --do-not-de-elevate on Windows: started by an app running as administrator
  it restarted itself as the user and the app lost hold of it
- checked on the win-ci Windows VM from install to uninstall (focus kept while typing, no
  taskbar button, no Chrome left, nothing left after uninstall) and on Linux with a real Chrome

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-14 02:43:32 +09:00
parent 56212c06dc
commit 1bd7defbb4
17 changed files with 884 additions and 43 deletions

View File

@@ -35,13 +35,17 @@ chrono = { version = "0.4", default-features = false, features = ["clock", "std"
zip = { version = "9", default-features = false, features = ["deflate"] }
unicode-normalization = "0.1"
[[test]]
name = "vdisplay_elevated"
harness = false
[profile.release]
lto = "thin"
codegen-units = 4
strip = "symbols"
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Globalization", "Win32_Security_Cryptography", "Win32_System_SystemInformation"] }
windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Globalization", "Win32_Security_Cryptography", "Win32_System_SystemInformation", "Win32_Devices_DeviceAndDriverInstallation", "Win32_Graphics_Gdi", "Win32_UI_WindowsAndMessaging", "Win32_System_Registry", "Win32_UI_Shell", "Win32_System_Threading"] }
[target.'cfg(unix)'.dependencies]
libc = "0.2"

View File

@@ -1,7 +1,8 @@
#!/bin/sh
# Build the Windows management app from Linux and lay it out ready to zip (admin.spec.md, "Windows build"):
# Jarvis/jarvis.exe, Jarvis/WebView2Loader.dll, Jarvis/python-src/{voice_worker,jarvis}
# Needs the x86_64-pc-windows-gnu Rust target, mingw-w64 and cmake (for Opus).
# Jarvis/jarvis.exe, Jarvis/WebView2Loader.dll, Jarvis/python-src/{voice_worker,jarvis},
# Jarvis/display-driver/ (Virtual Display Driver 25.7.23, MIT, SignPath-signed; "Virtual monitor")
# Needs the x86_64-pc-windows-gnu Rust target, mingw-w64, cmake (for Opus), curl and unzip.
# app/shell/package.sh <output folder>
set -eu
here=$(cd "$(dirname "$0")" && pwd)
@@ -17,4 +18,16 @@ for pkg in voice_worker jarvis; do
(cd "$here/../../src" && find "$pkg" -name __pycache__ -prune -o -name '*.spec.md' -prune -o -type f -print) |
while read -r f; do mkdir -p "$out/Jarvis/python-src/$(dirname "$f")"; cp "$here/../../src/$f" "$out/Jarvis/python-src/$f"; done
done
# The virtual monitor's driver, pinned by hash; its sample settings file is not used (the app
# writes its own).
vdd_url=https://github.com/VirtualDrivers/Virtual-Display-Driver/releases/download/25.7.23/VirtualDisplayDriver-x86.Driver.Only.zip
vdd_sha=e24210692b442b39af763536330ce78b423f19342b7a7792c26de3944e418b3a
vdd_zip="$target/vdd-25.7.23.zip"
[ -f "$vdd_zip" ] || curl -fsSL -o "$vdd_zip" "$vdd_url"
echo "$vdd_sha $vdd_zip" | sha256sum -c - >/dev/null || { rm -f "$vdd_zip"; echo "display driver hash mismatch" >&2; exit 1; }
tmp=$(mktemp -d "$target/vdd.XXXXXX")
unzip -q "$vdd_zip" -d "$tmp"
mkdir -p "$out/Jarvis/display-driver"
cp "$tmp/VirtualDisplayDriver/MttVDD.inf" "$tmp/VirtualDisplayDriver/mttvdd.cat" "$tmp/VirtualDisplayDriver/MttVDD.dll" "$out/Jarvis/display-driver/"
rm -rf "$tmp"
echo "$out/Jarvis"

View File

@@ -4,6 +4,7 @@
//! if that cannot be confirmed the window stays open and says so, and closing it again quits.
//!
//! jarvis [--location FILE]
//! jarvis --display-driver install|uninstall --result FILE (elevated, started by the app)
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
@@ -36,6 +37,10 @@ fn arg(name: &str) -> Option<String> {
}
fn main() {
// The elevated half of installing or removing the virtual monitor's driver: no window.
if let Some(code) = jarvis_app::vdisplay::elevated_main(&std::env::args().collect::<Vec<_>>()) {
std::process::exit(code);
}
let location = arg("--location").map(PathBuf::from).or_else(jarvis_app::datadir::default_location_file)
.expect("no location file path: pass --location");
// Log files go to the data folder once there is one; until then memory only.

View File

@@ -61,7 +61,9 @@ manages the MCP servers it connects.
- `features` (new section): `weather` (`getWeather`), `webSearch` (`webSearch`, `fetchWebPage`) and
`meals` (`logMeal`, `fetchMeals`, `deleteMeal`), each on by default; `collect` (background
collection, below; no brain tool) and `browser` (the bot browser, below "Bot browser", and its
brain tool `botBrowser`, `src/jarvis/tools/builtin/bot_browser.spec.md`), also on by default. The tools of a feature that
brain tool `botBrowser`, `src/jarvis/tools/builtin/bot_browser.spec.md`), also on by default;
`browserShare` (봇용 크롬창 화면공유 선택 표시, the bot browser on the virtual monitor, below
"Virtual monitor"), off by default. The tools of a feature that
is off go into the brain's `disabled_tools`: they are neither offered nor run
(`voice_worker.spec.md`, "Tools"). The other features of the 기능 screen (PC control, screen
viewing) come with their milestone (M7).
@@ -236,8 +238,8 @@ The LLM itself (Qwen3.6-35B-A3B, about 23 GB) is downloaded by Strata on its fir
## Bot browser (`app/src/browser.rs`)
`docs/v2/pc-features.md`, "백그라운드 크로미움으로 정보 수집" and "봇용 크롬창과 디스코드 화면공유". The
bot's own Chrome, in the data folder, driven over the Chrome DevTools Protocol (CDP). This part runs
it in the background (headless); the virtual monitor for screen sharing comes next (M6).
bot's own Chrome, in the data folder, driven over the Chrome DevTools Protocol (CDP). It runs in the
background (headless), or with `features.browserShare` in a window on the virtual monitor (below).
- `features.browser` (브라우저 제어, on by default). With it on and the `browser` step installed,
the services start the bot browser after the LLM and before the worker, and stop it after the
@@ -265,7 +267,48 @@ it in the background (headless); the virtual monitor for screen sharing comes ne
- Dashboard `browser`: `{state, note}` with `state` off, ready, failed (the note says why) or exited
(it ended on its own).
- Linux (development only): Ubuntu's AppArmor refuses Chrome's sandbox; `JARVIS_CHROME_NO_SANDBOX=1`
in the app's environment adds `--no-sandbox`. Windows always runs it sandboxed.
in the app's environment adds `--no-sandbox`. Windows always runs it sandboxed, with
`--do-not-de-elevate`: started by an app running as administrator, Chrome would otherwise start
again as the user and end the process the app holds, so the app could neither see nor stop it.
## Virtual monitor (`app/src/vdisplay.rs`)
`docs/v2/pc-features.md`, "봇용 크롬창과 디스코드 화면공유". With `features.browserShare` the bot
browser's window sits on a monitor that exists only for it, so the user can pick that monitor in
Discord's screen share ("화면" tab) while no real monitor shows the window. Windows only; elsewhere
the browser runs headless with a note.
- **Driver**: the open-source Virtual Display Driver 25.7.23 (MIT, signed by the SignPath
Foundation), its `MttVDD.inf`, `mttvdd.cat` and `MttVDD.dll` in `display-driver/` next to
`jarvis.exe` (`package.sh` downloads the release zip and checks its SHA-256). Installing and
removing need the administrator once: the app starts itself elevated (`jarvis.exe
--display-driver install|uninstall --result FILE`, Windows asks), waits, and reads why it failed
from FILE. Install writes the driver's settings to `%ProgramData%\JarvisBot\display-driver\vdd_settings.xml`
(the driver runs as a system service that cannot read the data folder, so this is the one place
outside it; readable by users and LOCAL SERVICE, writable only by the system and administrators),
points the driver at it (`HKLM\SOFTWARE\MikeTheTech\VirtualDisplayDriver` `VDDPATH`), creates the
device (`Root\MttVDD`, display class) and installs the driver; Windows then asks once whether to
install software from the SignPath Foundation. The new monitor is taken off the desktop at once.
The settings: one monitor, one mode (1280x720 at 30 Hz), no custom EDID. When another program's
copy of the same driver is installed (its `VDDPATH` is not ours) the app refuses to install or
remove and never touches its monitors. Uninstall removes the device, the driver package, the
registry key and the settings folder.
- **On only while used**: a service start first takes the monitor off the desktop (left over from a
run that ended badly). With `browserShare` it then attaches the monitor (no administrator needed)
right of and below the rightmost real monitor, touching it only at that monitor's bottom-right
corner, so the mouse hardly reaches it and no real monitor overlaps it. The stop closes the
browser first and then takes the monitor off, so Windows never moves the window onto a real
monitor.
- **The window**: Chrome starts with `--no-startup-window` (no headless), and the app opens its one
window over CDP in the background (`Target.createTarget` with a new window at the monitor's place
and size), so it never takes the focus from the game or program the user is in. The window
then becomes a tool window (not in Alt+Tab, Win+Tab or the taskbar), fills the monitor's work
area (Windows may give the monitor a taskbar of its own) and goes behind all other windows, all
without activating it. It keeps drawing when covered (`--disable-backgrounding-occluded-windows`,
`--disable-renderer-backgrounding`, `--disable-background-timer-throttling`). The window holds
the first tab, which the bot never closes; the bot's tabs open in it, the newest in front.
- If the monitor or the window cannot be made, the browser runs headless and the dashboard's
`browser` note says why ("가상 모니터 없이 실행: ...").
## LLM (`app/src/strata.rs`)
@@ -344,6 +387,7 @@ back as Korean messages for the screen. The Tauri shell passes the screens' `inv
| `discord_channel_info` `{id}` / `channel_sync_status` | a channel's Discord name and topic / each synced chat channel's last result (above) |
| `collect_results` `{target, limit}` | `{targets, items}`: every target folder with results, and the collected items newest first (`target` null: every target; `limit` 1-500, default 100) |
| `mcp_env_status` / `mcp_env_set` `{server, name, value}` | which MCP environment values are stored (`{server: {NAME: bool}}`, never the values) / store one (an empty value removes it); only for a saved server and name |
| `display_driver_status` / `display_driver_install` / `display_driver_uninstall` | the virtual monitor's driver: `{supported, installed, other}` (Windows only; `other`: another program's copy) / install or remove it elevated (above), then its status |
| `install_status` / `install_start` / `install_cancel` | the install steps and their state, bytes and last line / run what is not done in the background / stop it |
| `logs_query` `{tab, level, find, after, limit}` / `logs_error_report` / `logs_export` `{from, to}` | the log hub; the zip goes to `logs/` and its path comes back |
@@ -374,7 +418,8 @@ Plain HTML, CSS and an ES module, no build step, dark theme after `DESIGN.md`. A
with their switches; the bot screen also changes the token; the 기능 screen has the feature
switches and one card per MCP server (name, command, arguments and environment variable names
one per line, on/off, delete), and for a saved server a password box per environment variable
that stores its value on its own. The 백그라운드 수집 screen has the targets (a table: name, kind,
that stores its value on its own; on Windows also a card for the virtual monitor's driver (its
state and an install or remove button). The 백그라운드 수집 screen has the targets (a table: name, kind,
address or words, interval in minutes, on/off, delete), the keep days, and the results (the
newest 100, filtered by any target that has results, each with its time, title, link and the start of its text shown
as plain text).
@@ -402,8 +447,8 @@ URL), so other local programs cannot drive it.
runs the window says so; if it fails the window stays open with the reason, and closing it again
quits anyway. Quitting another way (the system shutting down) runs the same shutdown.
- `app/shell/package.sh <folder>` builds it on Linux (`x86_64-pc-windows-gnu`, mingw-w64) and lays
out `Jarvis/`: `jarvis.exe`, `WebView2Loader.dll` and `python-src/` (the `voice_worker` and
`jarvis` packages, no caches or specs). The worker's default sources are `python-src` next to the
out `Jarvis/`: `jarvis.exe`, `WebView2Loader.dll`, `python-src/` (the `voice_worker` and
`jarvis` packages, no caches or specs) and `display-driver/` (the virtual monitor's driver). The worker's default sources are `python-src` next to the
executable when it is there, else the repository's `src/`; the install reads its requirement
files from the same place. The installer (NSIS) comes later; the folder runs as it is.
@@ -449,4 +494,12 @@ downloads, install steps, logs; the Unix-only process tests excepted), and the w
wizard's folder, PC check and install (`uv` and the worker venv installed, a cancel leaving no
`uv` or Python process), the token stored DPAPI-encrypted, a start failing with its reason, the
log screen, error copy, zip export, and closing the window ending the process with nothing written
to `%APPDATA%` or `%LOCALAPPDATA%`.
to `%APPDATA%` or `%LOCALAPPDATA%`. The virtual monitor (`tests/vdisplay_elevated.rs`, no harness:
started elevated, the same program is the elevated half; `tests/bot_browser.rs` with a screen): the
driver installed through the elevated start (Windows' SignPath question answered once) with the
settings folder's permissions, the monitor off after install, on at the rightmost monitor's corner
while the bot browser runs and off after, one Chrome window there filling the work area and holding
the bot's tab, the focus staying in the program the user types into, no taskbar button, no Chrome
left after the stop, and the uninstall leaving no device, driver package, registry key or settings
folder. On Linux the same window test runs on an X display (one window at the given place and
size, the bot's tab in it).

View File

@@ -281,6 +281,18 @@ impl AdminApi {
Ok(json!({"set": !value.is_empty(),
"restartNeeded": self.controller()?.state() == crate::control::State::Running}))
}
// The virtual monitor's driver (admin.spec.md, "Virtual monitor"): installing or
// removing it runs this app elevated once; Windows asks the user.
"display_driver_status" => Ok(json!(crate::vdisplay::status())),
"display_driver_install" | "display_driver_uninstall" => {
let action = if command == "display_driver_install" { "install" } else { "uninstall" };
if let Err(e) = crate::vdisplay::run_elevated(action).await {
crate::warn_log!("admin", "Warning: display driver {action} failed: {e}");
return Err(e);
}
crate::info_log!("admin", "display driver {action} done");
Ok(json!(crate::vdisplay::status()))
}
"collect_results" => {
let target = args.get("target").and_then(Value::as_str).filter(|t| !t.is_empty());
let limit = args.get("limit").and_then(Value::as_u64).unwrap_or(100).clamp(1, 500) as usize;

View File

@@ -212,11 +212,11 @@ async fn refused(client: &mut TcpStream, why: &str) -> std::io::Result<()> {
// ---- Chrome ---------------------------------------------------------------------------------------
/// The arguments the bot's Chrome runs with (headless, its profile, CDP on a free port, every
/// connection through the proxy on `proxy_port`).
pub fn chrome_args(profile: &Path, proxy_port: u16, no_sandbox: bool) -> Vec<String> {
/// The arguments the bot's Chrome runs with (its profile, CDP on a free port, every connection
/// through the proxy on `proxy_port`): headless, or with `screen` a real window the app opens later
/// over CDP (admin.spec.md, "Virtual monitor").
pub fn chrome_args(profile: &Path, proxy_port: u16, no_sandbox: bool, screen: bool) -> Vec<String> {
let mut a = vec![
"--headless=new".to_owned(),
format!("--user-data-dir={}", profile.display()),
"--remote-debugging-port=0".into(),
format!("--proxy-server=http://127.0.0.1:{proxy_port}"),
@@ -235,7 +235,19 @@ pub fn chrome_args(profile: &Path, proxy_port: u16, no_sandbox: bool) -> Vec<Str
if no_sandbox {
a.push("--no-sandbox".into());
}
a.push("about:blank".into());
if cfg!(windows) {
// Started by an app running as administrator, Chrome would start again as the user and end
// the process the app holds, which could then neither see nor stop it.
a.push("--do-not-de-elevate".into());
}
if screen {
// A window opened at start would take the focus from the game the user is playing.
a.extend(["--no-startup-window", "--disable-backgrounding-occluded-windows", "--disable-renderer-backgrounding",
"--disable-background-timer-throttling"].map(String::from));
} else {
a.insert(0, "--headless=new".into());
a.push("about:blank".into());
}
a
}
@@ -246,15 +258,17 @@ pub struct BotBrowser {
}
impl BotBrowser {
/// Start Chrome `exe` with profile folder `profile` and wait until CDP answers.
pub async fn start(exe: &Path, profile: &Path) -> Result<Self, String> {
/// Start Chrome `exe` with profile folder `profile` and wait until CDP answers; with `screen`,
/// its one window sits there (the virtual monitor), opened without taking the focus and kept out
/// of Alt+Tab and the taskbar.
pub async fn start(exe: &Path, profile: &Path, screen: Option<crate::vdisplay::Rect>) -> Result<Self, String> {
let proxy = EgressProxy::start(is_public).await.map_err(|e| format!("proxy: {e}"))?;
std::fs::create_dir_all(profile).map_err(|e| e.to_string())?;
let port_file = profile.join("DevToolsActivePort");
let _ = std::fs::remove_file(&port_file);
let no_sandbox = cfg!(not(windows)) && std::env::var("JARVIS_CHROME_NO_SANDBOX").is_ok_and(|v| v == "1");
let mut cmd = tokio::process::Command::new(exe);
cmd.args(chrome_args(profile, proxy.port, no_sandbox)).stdin(std::process::Stdio::null())
cmd.args(chrome_args(profile, proxy.port, no_sandbox, screen.is_some())).stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::null()).stderr(std::process::Stdio::null()).kill_on_drop(true);
#[cfg(unix)]
cmd.process_group(0);
@@ -278,8 +292,35 @@ impl BotBrowser {
}
tokio::time::sleep(Duration::from_millis(200)).await;
};
crate::info_log!("browser", "bot browser ready (CDP on 127.0.0.1:{cdp_port})");
Ok(Self { child: Mutex::new(Some(child)), proxy, cdp_port })
let b = Self { child: Mutex::new(Some(child)), proxy, cdp_port };
if let Some(r) = screen {
if let Err(e) = b.open_window(r).await {
b.stop().await;
return Err(e);
}
}
crate::info_log!("browser", "bot browser ready (CDP on 127.0.0.1:{cdp_port}{})",
if screen.is_some() { ", on the virtual monitor" } else { "" });
Ok(b)
}
/// The window on `r`: a background page in a new window (CDP), then out of the window switcher
/// and fitted to the monitor's work area.
/// It holds the first tab, which the bot never closes, so the window stays.
async fn open_window(&self, r: crate::vdisplay::Rect) -> Result<(), String> {
let ws = version(self.cdp_port).await.and_then(|v| v["webSocketDebuggerUrl"].as_str().map(str::to_owned))
.ok_or("Chrome의 CDP 주소를 읽지 못했습니다")?;
cdp_call(&ws, serde_json::json!({"id": 1, "method": "Target.createTarget", "params": {"url": "about:blank",
"newWindow": true, "background": true, "left": r.x, "top": r.y, "width": r.w, "height": r.h}})).await
.map_err(|e| format!("Chrome 창을 열지 못했습니다: {e}"))?;
let pid = self.child.lock().unwrap().as_ref().and_then(|c| c.id()).unwrap_or(0);
for _ in 0..25 {
if cfg!(not(windows)) || crate::vdisplay::settle_windows(pid) > 0 {
return Ok(());
}
tokio::time::sleep(Duration::from_millis(200)).await;
}
Err("Chrome 창을 찾지 못했습니다".into())
}
pub fn cdp_url(&self) -> String {
@@ -323,6 +364,27 @@ async fn close_over_cdp(ws: &str) -> Result<(), String> {
.map_err(|e| e.to_string())
}
/// One CDP call on the browser connection `ws`; its result, or the error Chrome gave (5 s at most).
async fn cdp_call(ws: &str, msg: serde_json::Value) -> Result<serde_json::Value, String> {
use futures_util::{SinkExt, StreamExt};
let id = msg["id"].clone();
tokio::time::timeout(Duration::from_secs(5), async {
let (mut s, _) = tokio_tungstenite::connect_async(ws).await.map_err(|e| e.to_string())?;
s.send(tokio_tungstenite::tungstenite::Message::text(msg.to_string())).await.map_err(|e| e.to_string())?;
while let Some(m) = s.next().await {
let m = m.map_err(|e| e.to_string())?;
let Ok(v) = serde_json::from_str::<serde_json::Value>(m.to_text().unwrap_or("")) else { continue };
if v["id"] == id {
return match v.get("error") {
Some(e) => Err(e["message"].as_str().unwrap_or("error").to_owned()),
None => Ok(v["result"].clone()),
};
}
}
Err("connection closed".to_owned())
}).await.map_err(|_| "timeout".to_owned())?
}
async fn kill_tree(child: &mut tokio::process::Child) {
if let Some(pid) = child.id() {
#[cfg(unix)]
@@ -437,13 +499,27 @@ mod tests {
#[test]
fn chrome_goes_through_the_proxy_with_its_own_profile() {
let a = chrome_args(Path::new("/d/browser/profile"), 4321, false);
let a = chrome_args(Path::new("/d/browser/profile"), 4321, false, false);
for want in ["--headless=new", "--user-data-dir=/d/browser/profile", "--remote-debugging-port=0",
"--proxy-server=http://127.0.0.1:4321", "--proxy-bypass-list=<-loopback>", "--disable-quic"] {
assert!(a.iter().any(|x| x == want), "{want}");
}
assert!(!a.iter().any(|x| x == "--no-sandbox"));
assert!(chrome_args(Path::new("/p"), 1, true).iter().any(|x| x == "--no-sandbox"));
assert!(chrome_args(Path::new("/p"), 1, true, false).iter().any(|x| x == "--no-sandbox"));
assert_eq!(a.iter().any(|x| x == "--do-not-de-elevate"), cfg!(windows), "the app keeps hold of Chrome on Windows");
}
#[test]
fn on_a_screen_chrome_has_a_window_it_opens_later_and_keeps_drawing_when_covered() {
let a = chrome_args(Path::new("/p"), 4321, false, true);
assert!(!a.iter().any(|x| x.starts_with("--headless")));
// No window at start: the app opens it over CDP in the background, so it never takes focus.
assert!(a.iter().any(|x| x == "--no-startup-window"));
assert!(!a.iter().any(|x| x == "about:blank"));
for want in ["--disable-backgrounding-occluded-windows", "--disable-renderer-backgrounding",
"--proxy-server=http://127.0.0.1:4321", "--disable-quic"] {
assert!(a.iter().any(|x| x == want), "{want}");
}
}
}

View File

@@ -102,9 +102,9 @@ mod tests {
#[test]
fn features_that_are_off_turn_their_tools_off() {
let all_on = FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true };
let all_on = FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true, browser_share: true };
assert!(disabled_tools(&all_on).is_empty());
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false, collect: false, browser: false });
let off = disabled_tools(&FeatureSettings { weather: false, web_search: true, meals: false, collect: false, browser: false, browser_share: false });
assert_eq!(off, vec!["getWeather", "logMeal", "fetchMeals", "deleteMeal", "botBrowser"]);
}

View File

@@ -29,5 +29,6 @@ pub mod settings_edit;
pub mod speakers;
pub mod strata;
pub mod system;
pub mod vdisplay;
pub mod worker_client;
pub mod worker_process;

View File

@@ -30,7 +30,8 @@ pub enum BrowserRun {
/// Not started: the feature is off, or (with a note) it is not installed.
Off(Option<String>),
Failed(String),
Running(Arc<crate::browser::BotBrowser>),
/// Running; the note says why it runs without the virtual monitor that was asked for.
Running(Arc<crate::browser::BotBrowser>, Option<String>),
}
impl BrowserRun {
@@ -39,15 +40,18 @@ impl BrowserRun {
let (state, note) = match self {
BrowserRun::Off(note) => ("off", note.clone()),
BrowserRun::Failed(e) => ("failed", Some(e.clone())),
BrowserRun::Running(b) if b.running() => ("ready", None),
BrowserRun::Running(_) => ("exited", None),
BrowserRun::Running(b, note) if b.running() => ("ready", note.clone()),
BrowserRun::Running(_, note) => ("exited", note.clone()),
};
serde_json::json!({"state": state, "note": note})
}
/// Stop the browser, then take the virtual monitor off the desktop (its window first, so
/// Windows never moves it onto a real monitor).
pub async fn stop(&self) {
if let BrowserRun::Running(b) = self {
if let BrowserRun::Running(b, _) = self {
b.stop().await;
crate::vdisplay::detach();
}
}
}
@@ -63,9 +67,11 @@ impl Services {
}
}
/// Start the bot browser when its feature is on and it is installed; a failure leaves the services
/// running without it.
/// Start the bot browser when its feature is on and it is installed, on the virtual monitor when
/// sharing is on; a failure leaves the services running without it (or without the monitor).
async fn start_browser(data_dir: &Path, settings: &crate::settings::Settings) -> BrowserRun {
// A monitor an earlier run left on goes off first.
crate::vdisplay::detach();
if !settings.features.browser {
return BrowserRun::Off(None);
}
@@ -74,8 +80,24 @@ async fn start_browser(data_dir: &Path, settings: &crate::settings::Settings) ->
crate::warn_log!("browser", "the bot browser is not installed");
return BrowserRun::Off(Some("설치되지 않음".into()));
}
match crate::browser::BotBrowser::start(&layout.chrome_exe(), &crate::browser::profile_dir(data_dir)).await {
Ok(b) => BrowserRun::Running(Arc::new(b)),
let (exe, profile) = (layout.chrome_exe(), crate::browser::profile_dir(data_dir));
let mut note = None;
if settings.features.browser_share {
let started = match crate::vdisplay::attach() {
Ok(r) => crate::browser::BotBrowser::start(&exe, &profile, Some(r)).await,
Err(e) => Err(e),
};
match started {
Ok(b) => return BrowserRun::Running(Arc::new(b), None),
Err(e) => {
crate::vdisplay::detach();
crate::warn_log!("browser", "Warning: no virtual monitor for the bot browser: {e}");
note = Some(format!("가상 모니터 없이 실행: {e}"));
}
}
}
match crate::browser::BotBrowser::start(&exe, &profile, None).await {
Ok(b) => BrowserRun::Running(Arc::new(b), note),
Err(e) => {
crate::warn_log!("browser", "Warning: the bot browser did not start: {e}");
BrowserRun::Failed(e)
@@ -120,7 +142,7 @@ pub async fn start_services(data_dir: &Path, llm: Arc<Llm>) -> Result<Services,
}
wopts.env.insert("JARVIS_CONFIG_PATH".into(), brain_config.to_string_lossy().into_owned());
wopts.env.insert(crate::features::MCP_ENV_VAR.into(), mcp_env.to_string());
if let BrowserRun::Running(b) = &browser {
if let BrowserRun::Running(b, _) = &browser {
wopts.env.insert("JARVIS_BROWSER_CDP".into(), b.cdp_url());
}
// Background collection (admin.spec.md): the enabled targets, and the keep days in any case.

View File

@@ -95,6 +95,9 @@ pub struct FeatureSettings {
pub collect: bool,
/// The bot browser (admin.spec.md, "Bot browser").
pub browser: bool,
/// The bot browser's window on a virtual monitor to share in Discord (admin.spec.md, "Virtual
/// monitor"); off by default.
pub browser_share: bool,
}
/// Background collection (admin.spec.md, "Background collection"); no targets by default.
@@ -186,7 +189,7 @@ impl Default for Settings {
response: PolicySettings::default(),
memory: Map::new(),
llm: LlmSettings { url: None, port: 8080, context: 32768 },
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true },
features: FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true, browser_share: false },
mcp_servers: vec![],
collect: CollectSettings { keep_days: 30, targets: vec![] },
}
@@ -508,6 +511,7 @@ pub fn parse_settings(raw: &Value) -> Result<Settings, SettingsError> {
meals: get_bool(f, "meals", "features", d.features.meals)?,
collect: get_bool(f, "collect", "features", d.features.collect)?,
browser: get_bool(f, "browser", "features", d.features.browser)?,
browser_share: get_bool(f, "browserShare", "features", d.features.browser_share)?,
}
},
mcp_servers: parse_mcp_servers(raw)?,
@@ -807,13 +811,13 @@ mod tests {
#[test]
fn features_and_mcp_servers_are_read_and_checked() {
let s = parse_settings(&json!({}));
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true });
assert_eq!(s.unwrap().features, FeatureSettings { weather: true, web_search: true, meals: true, collect: true, browser: true, browser_share: false });
let s = parse_settings(&json!({
"features": {"weather": false, "meals": false},
"features": {"weather": false, "meals": false, "browserShare": true},
"mcpServers": [{"name": "brave", "command": "npx", "args": ["-y", "brave"], "env": ["BRAVE_API_KEY", " "]},
{"name": "time-2", "command": "uvx", "enabled": false}]
})).unwrap();
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false, collect: true, browser: true });
assert_eq!(s.features, FeatureSettings { weather: false, web_search: true, meals: false, collect: true, browser: true, browser_share: true });
assert_eq!(s.mcp_servers, vec![
McpServerSettings { name: "brave".into(), command: "npx".into(), args: vec!["-y".into(), "brave".into()],
env: vec!["BRAVE_API_KEY".into()], enabled: true },

View File

@@ -124,6 +124,7 @@ pub fn fields() -> Vec<Field> {
f("features.meals", Features, Bool, json!(true), "식단 기록"),
f("features.collect", Features, Bool, json!(true), "백그라운드 수집"),
f("features.browser", Features, Bool, json!(true), "브라우저 제어(봇용 크롬)"),
f("features.browserShare", Features, Bool, json!(false), "봇용 크롬창 화면공유 선택 표시(가상 모니터, Windows)"),
f("mcpServers", Features, McpServers, json!([]), "MCP 서버"),
// Background collection (admin.spec.md, "Background collection"): no targets by default.
f("collect.targets", Collect, CollectTargets, json!([]), "수집 대상"),

531
app/src/vdisplay.rs Normal file
View File

@@ -0,0 +1,531 @@
//! The virtual monitor the bot browser's window sits on, so the user can share it in Discord's
//! "Screen" tab while no real monitor shows it (admin.spec.md, "Virtual monitor";
//! `docs/v2/pc-features.md`, "봇용 크롬창과 디스코드 화면공유").
//!
//! The monitor comes from the open-source Virtual Display Driver (MIT, signed by the SignPath
//! Foundation), shipped next to `jarvis.exe` in `display-driver/`. Installing it needs the
//! administrator once (`--display-driver install`, started elevated by the app); turning the monitor
//! on and off does not: the app attaches it to the desktop only while the bot browser uses it.
use std::path::{Path, PathBuf};
use serde::Serialize;
/// The monitor's mode (the driver offers only this one): small enough to share cheaply.
pub const WIDTH: i32 = 1280;
pub const HEIGHT: i32 = 720;
pub const REFRESH: u32 = 30;
/// The adapter name the driver's INF gives its monitors.
pub const ADAPTER_NAME: &str = "Virtual Display Driver";
pub const HARDWARE_ID: &str = "Root\\MttVDD";
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
pub struct Rect {
pub x: i32,
pub y: i32,
pub w: i32,
pub h: i32,
}
/// Where the virtual monitor goes: right of and below the rightmost real monitor, touching it only
/// at that monitor's bottom-right corner, so the mouse hardly ever wanders onto it and no real
/// monitor overlaps it (nothing reaches further right).
pub fn corner(real: &[Rect]) -> (i32, i32) {
real.iter()
.max_by_key(|r| (r.x + r.w, r.y + r.h))
.map(|r| (r.x + r.w, r.y + r.h))
.unwrap_or((0, 0))
}
/// The driver's settings file: one monitor with one mode, the driver's other options at their
/// defaults.
pub fn settings_xml() -> String {
format!(
"<?xml version='1.0' encoding='utf-8'?>\r\n<vdd_settings>\r\n\
<monitors><count>1</count></monitors>\r\n\
<gpu><friendlyname>default</friendlyname></gpu>\r\n\
<global><g_refresh_rate>{REFRESH}</g_refresh_rate></global>\r\n\
<resolutions><resolution><width>{WIDTH}</width><height>{HEIGHT}</height><refresh_rate>{REFRESH}</refresh_rate></resolution></resolutions>\r\n\
<options><CustomEdid>false</CustomEdid><PreventSpoof>false</PreventSpoof><EdidCeaOverride>false</EdidCeaOverride>\
<HardwareCursor>true</HardwareCursor><SDR10bit>false</SDR10bit><HDRPlus>false</HDRPlus>\
<logging>false</logging><debuglogging>false</debuglogging></options>\r\n</vdd_settings>\r\n"
)
}
/// Where the driver reads its settings: the driver runs as a system service, which cannot read a
/// data folder in the user's profile, so this is the one place outside the data folder.
pub fn config_dir() -> Option<PathBuf> {
std::env::var_os("ProgramData").map(|p| PathBuf::from(p).join("JarvisBot").join("display-driver"))
}
/// The driver files shipped with the app.
pub fn driver_dir() -> Option<PathBuf> {
std::env::current_exe().ok()?.parent().map(|d| d.join("display-driver"))
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
pub struct DriverStatus {
/// Windows only.
pub supported: bool,
/// Installed by this app (its settings folder is the driver's).
pub installed: bool,
/// Another program's copy of the same driver is installed; the app leaves it alone.
pub other: bool,
}
#[cfg(not(windows))]
pub fn status() -> DriverStatus {
DriverStatus { supported: false, installed: false, other: false }
}
#[cfg(not(windows))]
pub fn attach() -> Result<Rect, String> {
Err("가상 모니터는 Windows에서만 씁니다".into())
}
#[cfg(not(windows))]
pub fn detach() {}
#[cfg(not(windows))]
pub fn settle_windows(_pid: u32) -> usize {
0
}
#[cfg(not(windows))]
pub async fn run_elevated(_action: &str) -> Result<(), String> {
Err("가상 모니터는 Windows에서만 씁니다".into())
}
/// `jarvis.exe --display-driver install|uninstall --result FILE`: the elevated half of
/// `run_elevated`; the outcome (empty, or why it failed) goes to FILE. `None` without the option.
pub fn elevated_main(args: &[String]) -> Option<i32> {
let i = args.iter().position(|a| a == "--display-driver")?;
let action = args.get(i + 1).map(String::as_str).unwrap_or("");
let result = args.iter().position(|a| a == "--result").and_then(|j| args.get(j + 1)).map(PathBuf::from);
let outcome = match (action, driver_dir()) {
(_, None) => Err("앱 폴더를 찾지 못했습니다".to_owned()),
("install", Some(d)) => install(&d),
("uninstall", Some(d)) => uninstall(&d),
_ => Err(format!("알 수 없는 동작: {action}")),
};
if let Some(f) = result {
let _ = std::fs::write(f, outcome.as_ref().err().map(String::as_str).unwrap_or(""));
}
Some(if outcome.is_ok() { 0 } else { 1 })
}
#[cfg(not(windows))]
fn install(_driver: &Path) -> Result<(), String> {
Err("가상 모니터는 Windows에서만 씁니다".into())
}
#[cfg(not(windows))]
fn uninstall(_driver: &Path) -> Result<(), String> {
Err("가상 모니터는 Windows에서만 씁니다".into())
}
#[cfg(windows)]
pub use win::{attach, detach, run_elevated, settle_windows, status};
#[cfg(windows)]
use win::{install, uninstall};
#[cfg(windows)]
mod win {
use super::*;
use std::os::windows::ffi::OsStrExt;
use windows_sys::core::BOOL;
use windows_sys::Win32::Devices::DeviceAndDriverInstallation::*;
use windows_sys::Win32::Foundation::*;
use windows_sys::Win32::Graphics::Gdi::*;
use windows_sys::Win32::System::Registry::{RegGetValueW, HKEY_LOCAL_MACHINE, RRF_RT_REG_SZ};
use windows_sys::Win32::System::Threading::{GetExitCodeProcess, WaitForSingleObject, INFINITE};
use windows_sys::Win32::UI::Shell::{ShellExecuteExW, SEE_MASK_NOCLOSEPROCESS, SHELLEXECUTEINFOW};
use windows_sys::Win32::UI::WindowsAndMessaging::*;
const REG_KEY: &str = "SOFTWARE\\MikeTheTech\\VirtualDisplayDriver";
fn wide(s: &str) -> Vec<u16> {
s.encode_utf16().chain(Some(0)).collect()
}
fn wide_path(p: &Path) -> Vec<u16> {
p.as_os_str().encode_wide().chain(Some(0)).collect()
}
fn text(buf: &[u16]) -> String {
String::from_utf16_lossy(&buf[..buf.iter().position(|&c| c == 0).unwrap_or(buf.len())])
}
struct Device {
name: [u16; 32],
string: String,
attached: bool,
}
fn devices() -> Vec<Device> {
let mut out = vec![];
for i in 0.. {
let mut dd = DISPLAY_DEVICEW { cb: std::mem::size_of::<DISPLAY_DEVICEW>() as u32, ..Default::default() };
// SAFETY: `dd` is a DISPLAY_DEVICEW with its size set, as the call requires.
if unsafe { EnumDisplayDevicesW(std::ptr::null(), i, &mut dd, 0) } == 0 {
break;
}
out.push(Device { name: dd.DeviceName, string: text(&dd.DeviceString),
attached: dd.StateFlags & DISPLAY_DEVICE_ATTACHED_TO_DESKTOP != 0 });
}
out
}
fn current(name: &[u16; 32]) -> Option<Rect> {
let mut dm = DEVMODEW { dmSize: std::mem::size_of::<DEVMODEW>() as u16, ..Default::default() };
// SAFETY: `name` is NUL-terminated (from EnumDisplayDevicesW) and `dm` has its size set.
if unsafe { EnumDisplaySettingsExW(name.as_ptr(), ENUM_CURRENT_SETTINGS, &mut dm, 0) } == 0 {
return None;
}
// SAFETY: for a display, the position variant of the union is the one Windows fills.
let pos = unsafe { dm.Anonymous1.Anonymous2.dmPosition };
Some(Rect { x: pos.x, y: pos.y, w: dm.dmPelsWidth as i32, h: dm.dmPelsHeight as i32 })
}
/// Put the display at `rect`, or take it off the desktop with `None`, and keep that.
fn set_mode(name: &[u16; 32], rect: Option<Rect>) -> Result<(), String> {
let mut dm = DEVMODEW { dmSize: std::mem::size_of::<DEVMODEW>() as u16, ..Default::default() };
// SAFETY: as in `current`; a failure leaves `dm` zeroed, which the fields below complete.
unsafe { EnumDisplaySettingsExW(name.as_ptr(), ENUM_REGISTRY_SETTINGS, &mut dm, 0) };
let r = rect.unwrap_or(Rect { x: 0, y: 0, w: 0, h: 0 });
dm.dmFields = DM_POSITION | DM_PELSWIDTH | DM_PELSHEIGHT;
dm.Anonymous1.Anonymous2 = DEVMODEW_0_1 { dmPosition: POINTL { x: r.x, y: r.y }, ..Default::default() };
dm.dmPelsWidth = r.w as u32;
dm.dmPelsHeight = r.h as u32;
// SAFETY: valid NUL-terminated device name and DEVMODEW; null window and parameter.
let changed = unsafe {
ChangeDisplaySettingsExW(name.as_ptr(), &dm, std::ptr::null_mut(), CDS_UPDATEREGISTRY | CDS_NORESET, std::ptr::null())
};
// SAFETY: the documented "apply the stored changes" call: all null, no flags.
let applied = unsafe { ChangeDisplaySettingsExW(std::ptr::null(), std::ptr::null(), std::ptr::null_mut(), 0, std::ptr::null()) };
if changed != DISP_CHANGE_SUCCESSFUL || applied != DISP_CHANGE_SUCCESSFUL {
return Err(format!("화면 설정을 바꾸지 못했습니다 ({changed}, {applied})"));
}
Ok(())
}
fn vdd_path() -> Option<String> {
let mut buf = [0u16; 520];
let mut size = (buf.len() * 2) as u32;
let (key, value) = (wide(REG_KEY), wide("VDDPATH"));
// SAFETY: valid NUL-terminated strings; `buf` holds `size` bytes.
let r = unsafe {
RegGetValueW(HKEY_LOCAL_MACHINE, key.as_ptr(), value.as_ptr(), RRF_RT_REG_SZ, std::ptr::null_mut(),
buf.as_mut_ptr().cast(), &mut size)
};
(r == ERROR_SUCCESS).then(|| text(&buf))
}
fn ours() -> bool {
match (vdd_path(), config_dir()) {
(Some(p), Some(c)) => p.trim_end_matches('\\').eq_ignore_ascii_case(c.to_string_lossy().trim_end_matches('\\')),
_ => false,
}
}
fn adapter() -> Option<Device> {
devices().into_iter().find(|d| d.string == ADAPTER_NAME)
}
pub fn status() -> DriverStatus {
let present = adapter().is_some() || !hardware_devices(false).is_empty();
let ours = ours();
DriverStatus { supported: true, installed: ours && present, other: present && !ours }
}
/// Put the virtual monitor on the desktop at `corner` and return where it is.
pub fn attach() -> Result<Rect, String> {
if !ours() {
return Err("가상 모니터 드라이버가 설치되지 않았습니다".into());
}
let v = adapter().ok_or("가상 모니터를 찾지 못했습니다")?;
let real: Vec<Rect> = devices().iter().filter(|d| d.attached && d.string != ADAPTER_NAME)
.filter_map(|d| current(&d.name)).collect();
let (x, y) = corner(&real);
set_mode(&v.name, Some(Rect { x, y, w: WIDTH, h: HEIGHT }))?;
current(&v.name).filter(|r| r.w > 0).ok_or_else(|| "가상 모니터가 켜지지 않았습니다".into())
}
/// Take the virtual monitor off the desktop (when it is ours and on it).
pub fn detach() {
if !ours() {
return;
}
if let Some(v) = adapter().filter(|d| d.attached) {
if let Err(e) = set_mode(&v.name, None) {
crate::warn_log!("browser", "Warning: the virtual monitor stays on: {e}");
}
}
}
struct Found {
pid: u32,
windows: Vec<HWND>,
}
unsafe extern "system" fn collect(hwnd: HWND, lparam: LPARAM) -> BOOL {
// SAFETY: `lparam` is the `Found` that `settle_windows` passes and outlives EnumWindows.
let found = unsafe { &mut *(lparam as *mut Found) };
let mut pid = 0u32;
let mut class = [0u16; 64];
// SAFETY: plain queries on a window handle EnumWindows gave; buffers are sized as passed.
unsafe {
GetWindowThreadProcessId(hwnd, &mut pid);
if pid == found.pid && IsWindowVisible(hwnd) != 0 && GetWindow(hwnd, GW_OWNER).is_null()
&& GetClassNameW(hwnd, class.as_mut_ptr(), class.len() as i32) > 0 && text(&class) == "Chrome_WidgetWin_1" {
found.windows.push(hwnd);
}
}
1
}
/// Make process `pid`'s top-level Chrome windows tool windows (not in Alt+Tab, Win+Tab or the
/// taskbar), fill their monitor's work area (Windows may give the virtual monitor a taskbar of
/// its own, which would cover the page's bottom) and put them behind all others, all without
/// activating them. Returns how many it made tool windows.
pub fn settle_windows(pid: u32) -> usize {
let mut found = Found { pid, windows: vec![] };
// SAFETY: `collect` only reads windows and writes into `found`, which lives through the call.
unsafe { EnumWindows(Some(collect), &mut found as *mut Found as LPARAM) };
let mut changed = 0;
for h in found.windows {
// SAFETY: style changes on a window of our own child process; hiding and showing it
// again is what makes the taskbar notice the new style.
unsafe {
let ex = GetWindowLongPtrW(h, GWL_EXSTYLE) as u32;
if ex & WS_EX_TOOLWINDOW == 0 {
ShowWindow(h, SW_HIDE);
SetWindowLongPtrW(h, GWL_EXSTYLE, ((ex | WS_EX_TOOLWINDOW) & !WS_EX_APPWINDOW) as isize);
ShowWindow(h, SW_SHOWNOACTIVATE);
changed += 1;
}
let mut mi = MONITORINFO { cbSize: std::mem::size_of::<MONITORINFO>() as u32, ..Default::default() };
if GetMonitorInfoW(MonitorFromWindow(h, MONITOR_DEFAULTTONEAREST), &mut mi) != 0 {
let w = mi.rcWork;
SetWindowPos(h, HWND_BOTTOM, w.left, w.top, w.right - w.left, w.bottom - w.top, SWP_NOACTIVATE);
} else {
SetWindowPos(h, HWND_BOTTOM, 0, 0, 0, 0, SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE);
}
}
}
changed
}
/// The display-class devices with the driver's hardware ID (`present`: only those plugged in).
fn hardware_devices(present: bool) -> Vec<SP_DEVINFO_DATA> {
let (set, list) = hardware_set(present);
if set != INVALID_HANDLE_VALUE as HDEVINFO {
// SAFETY: the list from `hardware_set`, no longer used.
unsafe { SetupDiDestroyDeviceInfoList(set) };
}
list
}
fn hardware_set(present: bool) -> (HDEVINFO, Vec<SP_DEVINFO_DATA>) {
// SAFETY: a device list for the display class; destroyed by the callers that keep it.
let set = unsafe { SetupDiGetClassDevsW(&GUID_DEVCLASS_DISPLAY, std::ptr::null(), std::ptr::null_mut(), if present { DIGCF_PRESENT } else { 0 }) };
let mut out = vec![];
if set == INVALID_HANDLE_VALUE as HDEVINFO {
return (set, out);
}
for i in 0.. {
let mut di = SP_DEVINFO_DATA { cbSize: std::mem::size_of::<SP_DEVINFO_DATA>() as u32, ..Default::default() };
// SAFETY: `di` has its size set; the loop ends at the first index past the list.
if unsafe { SetupDiEnumDeviceInfo(set, i, &mut di) } == 0 {
break;
}
let mut buf = [0u16; 256];
// SAFETY: `buf` holds the byte count passed.
let ok = unsafe {
SetupDiGetDeviceRegistryPropertyW(set, &di, SPDRP_HARDWAREID, std::ptr::null_mut(), buf.as_mut_ptr().cast(),
(buf.len() * 2) as u32, std::ptr::null_mut())
};
if ok != 0 && text(&buf).eq_ignore_ascii_case(HARDWARE_ID) {
out.push(di);
}
}
(set, out)
}
fn run(cmd: &str, args: &[&str]) -> Result<(), String> {
use std::os::windows::process::CommandExt;
let s = std::process::Command::new(cmd).args(args).creation_flags(0x0800_0000)
.stdout(std::process::Stdio::null()).stderr(std::process::Stdio::null()).status()
.map_err(|e| format!("{cmd}: {e}"))?;
if s.success() { Ok(()) } else { Err(format!("{cmd} failed ({s})")) }
}
/// Elevated: settings folder, the driver's registry value, the device, and its driver. Windows
/// asks the user once whether to install software from the SignPath Foundation.
pub(super) fn install(driver: &Path) -> Result<(), String> {
let inf = driver.join("MttVDD.inf");
if !inf.is_file() {
return Err("드라이버 파일이 없습니다 (display-driver\\MttVDD.inf)".into());
}
let (present, ours) = (adapter().is_some() || !hardware_devices(false).is_empty(), ours());
if present && !ours {
return Err("다른 프로그램이 설치한 가상 디스플레이 드라이버가 이미 있습니다".into());
}
let dir = config_dir().ok_or("ProgramData 폴더를 찾지 못했습니다")?;
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
std::fs::write(dir.join("vdd_settings.xml"), settings_xml()).map_err(|e| e.to_string())?;
// Only the system, administrators, the driver's service account (LOCAL SERVICE) and users
// reading: no user can change what the driver reads.
let d = dir.to_string_lossy().into_owned();
run("icacls", &[&d, "/inheritance:r", "/grant:r", "*S-1-5-18:(OI)(CI)F", "*S-1-5-32-544:(OI)(CI)F",
"*S-1-5-19:(OI)(CI)RX", "*S-1-5-32-545:(OI)(CI)RX"])?;
run("reg", &["add", &format!("HKLM\\{REG_KEY}"), "/v", "VDDPATH", "/t", "REG_SZ", "/d", &d, "/f"])?;
if hardware_devices(false).is_empty() {
create_device()?;
}
let (hw, inf_w) = (wide(HARDWARE_ID), wide_path(&inf));
let mut reboot: BOOL = 0;
// SAFETY: valid NUL-terminated strings; no parent window.
if unsafe { UpdateDriverForPlugAndPlayDevicesW(std::ptr::null_mut(), hw.as_ptr(), inf_w.as_ptr(), INSTALLFLAG_FORCE, &mut reboot) } == 0 {
// SAFETY: plain call.
let e = unsafe { GetLastError() };
return Err(if e == ERROR_CANCELLED { "드라이버 설치를 취소했습니다".into() } else { format!("드라이버를 설치하지 못했습니다 ({e})") });
}
// The new monitor joins the desktop at once; it stays off until the bot browser uses it.
for _ in 0..50 {
if adapter().is_some_and(|a| a.attached) {
detach();
break;
}
std::thread::sleep(std::time::Duration::from_millis(200));
}
Ok(())
}
fn create_device() -> Result<(), String> {
let (name, hw) = (wide("Display"), {
let mut v: Vec<u16> = HARDWARE_ID.encode_utf16().collect();
v.extend([0, 0]);
v
});
// SAFETY: the devcon "install" sequence: a list, one generated device with the hardware ID,
// registered with the class installer; the list is destroyed on every path.
unsafe {
let set = SetupDiCreateDeviceInfoList(&GUID_DEVCLASS_DISPLAY, std::ptr::null_mut());
if set == INVALID_HANDLE_VALUE as HDEVINFO {
return Err(format!("장치 목록을 만들지 못했습니다 ({})", GetLastError()));
}
let mut di = SP_DEVINFO_DATA { cbSize: std::mem::size_of::<SP_DEVINFO_DATA>() as u32, ..Default::default() };
let ok = SetupDiCreateDeviceInfoW(set, name.as_ptr(), &GUID_DEVCLASS_DISPLAY, std::ptr::null(), std::ptr::null_mut(), DICD_GENERATE_ID, &mut di) != 0
&& SetupDiSetDeviceRegistryPropertyW(set, &mut di, SPDRP_HARDWAREID, hw.as_ptr().cast(), (hw.len() * 2) as u32) != 0
&& SetupDiCallClassInstaller(DIF_REGISTERDEVICE, set, &di) != 0;
let e = GetLastError();
SetupDiDestroyDeviceInfoList(set);
if ok { Ok(()) } else { Err(format!("가상 모니터 장치를 만들지 못했습니다 ({e})")) }
}
}
/// Elevated: remove the device, the driver, the registry value and the settings folder; only
/// when this app installed it.
pub(super) fn uninstall(driver: &Path) -> Result<(), String> {
if !ours() {
return Err("이 앱이 설치한 가상 디스플레이 드라이버가 아닙니다".into());
}
let (set, list) = hardware_set(false);
for di in &list {
// SAFETY: removing a device of the list `set`, which is destroyed after the loop.
unsafe { SetupDiCallClassInstaller(DIF_REMOVE, set, di) };
}
if set != INVALID_HANDLE_VALUE as HDEVINFO {
// SAFETY: the list from `hardware_set`.
unsafe { SetupDiDestroyDeviceInfoList(set) };
}
let inf = wide_path(&driver.join("MttVDD.inf"));
let mut reboot: BOOL = 0;
// SAFETY: valid NUL-terminated path; a driver not in the store is just a failure we ignore.
unsafe { DiUninstallDriverW(std::ptr::null_mut(), inf.as_ptr(), 0, &mut reboot) };
let _ = run("reg", &["delete", &format!("HKLM\\{REG_KEY}"), "/f"]);
if let Some(dir) = config_dir() {
let _ = std::fs::remove_dir_all(&dir);
if let Some(parent) = dir.parent() {
let _ = std::fs::remove_dir(parent);
}
}
if hardware_devices(false).is_empty() { Ok(()) } else { Err("가상 모니터 장치를 지우지 못했습니다".into()) }
}
/// Run `jarvis.exe --display-driver <action>` as administrator (Windows asks the user) and wait.
pub async fn run_elevated(action: &str) -> Result<(), String> {
let exe = std::env::current_exe().map_err(|e| e.to_string())?;
let result = std::env::temp_dir().join(format!("jarvis-display-{}.txt", std::process::id()));
let _ = std::fs::remove_file(&result);
let params = format!("--display-driver {action} --result \"{}\"", result.display());
let code = tokio::task::spawn_blocking(move || -> Result<u32, String> {
let (verb, file, params) = (wide("runas"), wide_path(&exe), wide(&params));
// SAFETY: SHELLEXECUTEINFOW is plain data; all zero is its documented empty state.
let mut info: SHELLEXECUTEINFOW = unsafe { std::mem::zeroed() };
info.cbSize = std::mem::size_of::<SHELLEXECUTEINFOW>() as u32;
info.fMask = SEE_MASK_NOCLOSEPROCESS;
info.lpVerb = verb.as_ptr();
info.lpFile = file.as_ptr();
info.lpParameters = params.as_ptr();
info.nShow = SW_HIDE;
// SAFETY: `info` is filled as documented and its strings live through the call; the
// process handle it returns is waited on and closed here.
unsafe {
if ShellExecuteExW(&mut info) == 0 {
let e = GetLastError();
return Err(if e == ERROR_CANCELLED { "관리자 권한을 받지 못했습니다".into() } else { format!("관리자 권한으로 실행하지 못했습니다 ({e})") });
}
WaitForSingleObject(info.hProcess, INFINITE);
let mut code = 1u32;
GetExitCodeProcess(info.hProcess, &mut code);
CloseHandle(info.hProcess);
Ok(code)
}
}).await.map_err(|e| e.to_string())??;
let why = std::fs::read_to_string(&result).unwrap_or_default();
let _ = std::fs::remove_file(&result);
match code {
0 => Ok(()),
_ if !why.is_empty() => Err(why),
c => Err(format!("드라이버 작업이 실패했습니다 (종료 코드 {c})")),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_monitor_touches_the_rightmost_monitor_only_at_its_corner() {
let one = [Rect { x: 0, y: 0, w: 1920, h: 1080 }];
assert_eq!(corner(&one), (1920, 1080));
// A second monitor left of the main one, and one above-right: the rightmost decides.
let three = [Rect { x: 0, y: 0, w: 2560, h: 1440 }, Rect { x: -1920, y: 0, w: 1920, h: 1080 },
Rect { x: 2560, y: -400, w: 1080, h: 1920 }];
let (x, y) = corner(&three);
assert_eq!((x, y), (3640, 1520));
for r in &three {
assert!(r.x + r.w <= x, "no real monitor reaches into the virtual one");
}
// Two equally far right: the lower corner, so neither overlaps.
let stacked = [Rect { x: 0, y: 0, w: 1920, h: 1080 }, Rect { x: 0, y: 1080, w: 1920, h: 1080 }];
assert_eq!(corner(&stacked), (1920, 2160));
}
#[test]
fn the_driver_gets_one_monitor_with_the_one_mode() {
let x = settings_xml();
assert!(x.contains("<count>1</count>"));
assert!(x.contains(&format!("<width>{WIDTH}</width><height>{HEIGHT}</height><refresh_rate>{REFRESH}</refresh_rate>")));
assert_eq!(x.matches("<resolution>").count(), 1);
assert!(x.contains("<CustomEdid>false</CustomEdid>"), "the driver never loads an EDID file from its folder");
}
#[test]
fn only_the_display_driver_option_runs_the_elevated_half() {
assert_eq!(elevated_main(&["jarvis".into(), "--location".into(), "x".into()]), None);
}
}

View File

@@ -36,7 +36,7 @@ async fn the_bot_browser_reaches_public_pages_never_this_pc_and_stops() {
});
let profile = std::env::temp_dir().join(format!("jarvis-bot-browser-{}", std::process::id()));
let b = BotBrowser::start(std::path::Path::new(&exe), &profile).await.expect("chrome starts");
let b = BotBrowser::start(std::path::Path::new(&exe), &profile, None).await.expect("chrome starts");
assert!(b.running());
let client = reqwest::Client::builder().no_proxy().build().unwrap();
for url in [format!("http://127.0.0.1:{local}/"), format!("http://localhost:{local}/"), "https://example.com/".into()] {
@@ -80,8 +80,80 @@ async fn the_browser_step_installs_a_chrome_that_starts() {
assert!(installer.done("browser"));
assert!(!data.join("cache").join("downloads").read_dir().map(|mut d| d.next().is_some()).unwrap_or(false),
"the archive is removed after unpacking");
let b = BotBrowser::start(&layout.chrome_exe(), &jarvis_app::browser::profile_dir(&data)).await.expect("starts");
let b = BotBrowser::start(&layout.chrome_exe(), &jarvis_app::browser::profile_dir(&data), None).await.expect("starts");
assert!(b.running());
b.stop().await;
let _ = std::fs::remove_dir_all(&data);
}
/// With a screen, Chrome starts without a window and the app opens exactly one, at the screen's
/// place and size; the page tab the bot opens joins it. Needs a display on Linux (`DISPLAY`); on
/// Windows the screen is the virtual monitor (its driver installed), on the desktop only meanwhile.
#[tokio::test]
async fn on_a_screen_the_bot_browser_has_one_window_at_the_monitor() {
let Ok(exe) = std::env::var("JARVIS_TEST_CHROME") else {
eprintln!("skipped: set JARVIS_TEST_CHROME");
return;
};
if cfg!(unix) && std::env::var_os("DISPLAY").is_none() {
eprintln!("skipped: no DISPLAY");
return;
}
let profile = std::env::temp_dir().join(format!("jarvis-bot-browser-screen-{}", std::process::id()));
let screen = if cfg!(windows) {
let r = jarvis_app::vdisplay::attach().expect("the virtual monitor goes on");
eprintln!("virtual monitor at {r:?}");
r
} else {
jarvis_app::vdisplay::Rect { x: 40, y: 30, w: 1280, h: 720 }
};
let b = BotBrowser::start(std::path::Path::new(&exe), &profile, Some(screen)).await.expect("chrome starts");
let client = reqwest::Client::builder().no_proxy().build().unwrap();
let pages = |l: serde_json::Value| l.as_array().unwrap().iter().filter(|t| t["type"] == "page")
.map(|t| t["id"].as_str().unwrap().to_owned()).collect::<Vec<_>>();
let list: serde_json::Value = client.get(format!("http://127.0.0.1:{}/json/list", b.cdp_port)).send().await.unwrap()
.text().await.map(|t| serde_json::from_str(&t).unwrap()).unwrap();
let first = pages(list);
assert_eq!(first.len(), 1, "one page in one window");
client.put(format!("http://127.0.0.1:{}/json/new?about:blank", b.cdp_port)).send().await.unwrap();
// Each page's window over CDP: both tabs share the one window, at the screen's rectangle.
let ws: serde_json::Value = client.get(format!("http://127.0.0.1:{}/json/version", b.cdp_port)).send().await.unwrap()
.text().await.map(|t| serde_json::from_str(&t).unwrap()).unwrap();
let (mut s, _) = tokio_tungstenite::connect_async(ws["webSocketDebuggerUrl"].as_str().unwrap()).await.unwrap();
let list: serde_json::Value = client.get(format!("http://127.0.0.1:{}/json/list", b.cdp_port)).send().await.unwrap()
.text().await.map(|t| serde_json::from_str(&t).unwrap()).unwrap();
let ids = pages(list);
assert_eq!(ids.len(), 2);
let mut windows = std::collections::HashSet::new();
for (i, id) in ids.iter().enumerate() {
use futures_util::{SinkExt, StreamExt};
let msg = serde_json::json!({"id": i + 1, "method": "Browser.getWindowForTarget", "params": {"targetId": id}});
s.send(tokio_tungstenite::tungstenite::Message::text(msg.to_string())).await.unwrap();
let r = loop {
let v: serde_json::Value = serde_json::from_str(s.next().await.unwrap().unwrap().to_text().unwrap()).unwrap();
if v["id"] == i + 1 { break v["result"].clone(); }
};
eprintln!("window: {r}");
windows.insert(r["windowId"].as_i64().unwrap());
let b = &r["bounds"];
let (left, top, width, height) = (b["left"].as_i64().unwrap(), b["top"].as_i64().unwrap(),
b["width"].as_i64().unwrap(), b["height"].as_i64().unwrap());
let (x, y, w, h) = (screen.x as i64, screen.y as i64, screen.w as i64, screen.h as i64);
if cfg!(windows) {
// Inside the monitor, filling it but for the monitor's own taskbar if Windows gave it one.
assert!(left >= x && top >= y && left + width <= x + w && top + height <= y + h, "{b}");
assert!(width * height * 10 >= w * h * 8, "{b}");
} else {
assert_eq!((left, top, width, height), (x, y, w, h));
}
}
assert_eq!(windows.len(), 1, "the bot's tab opens in the same window");
if std::env::var_os("JARVIS_TEST_HOLD").is_some() {
// Time for a look at the screen (the Windows check takes a screenshot meanwhile).
tokio::time::sleep(std::time::Duration::from_secs(8)).await;
}
b.stop().await;
jarvis_app::vdisplay::detach();
let _ = std::fs::remove_dir_all(&profile);
}

View File

@@ -0,0 +1,23 @@
//! Installing and removing the virtual monitor's driver through the app's elevated half
//! (admin.spec.md, "Virtual monitor"). Windows only, with the driver files in `display-driver/` next
//! to this program and `JARVIS_TEST_VDISPLAY=install` or `uninstall`; Windows asks for the
//! administrator (and on install whether to trust the SignPath Foundation). Elsewhere it skips.
//! No test harness: started elevated, this same program is the elevated half.
fn main() {
let args: Vec<String> = std::env::args().collect();
if let Some(code) = jarvis_app::vdisplay::elevated_main(&args) {
std::process::exit(code);
}
let Ok(action) = std::env::var("JARVIS_TEST_VDISPLAY") else {
eprintln!("skipped: set JARVIS_TEST_VDISPLAY=install|uninstall");
return;
};
let rt = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap();
let outcome = rt.block_on(jarvis_app::vdisplay::run_elevated(&action));
let status = jarvis_app::vdisplay::status();
println!("{action}: {outcome:?} -> {status:?}");
outcome.expect("the elevated half succeeds");
assert_eq!(status.installed, action == "install");
assert!(!status.other);
}

View File

@@ -385,6 +385,28 @@ async function tokenCard() {
} }, "확인 후 저장")), out);
}
/** The virtual monitor's driver for "봇용 크롬창 화면공유 선택 표시" (Windows only). */
async function displayDriverCard() {
const st = await api("display_driver_status");
if (!st.supported) return null;
const label = h("span");
const btn = h("button", { class: "btn" });
const show = (s) => {
label.textContent = s.installed ? "설치됨" : s.other ? "다른 프로그램의 같은 드라이버가 있어 쓰지 않습니다" : "설치되지 않음";
btn.textContent = s.installed ? "드라이버 제거" : "드라이버 설치";
btn.disabled = s.other;
btn.onclick = async () => {
btn.disabled = true;
try { show(await api(s.installed ? "display_driver_uninstall" : "display_driver_install")); toast("완료했습니다"); }
catch (e) { fail(e); show(await api("display_driver_status")); }
};
};
show(st);
return h("div", { class: "card" }, h("h2", {}, "가상 모니터 드라이버"),
h("p", { class: "small muted" }, "봇용 크롬창을 실제 모니터에는 보이지 않는 가상 모니터에 두어, 디스코드 화면 공유의 \"화면\" 탭에서 고를 수 있게 합니다. 설치할 때 관리자 권한과 Windows의 장치 소프트웨어 설치 확인(SignPath Foundation)을 한 번 묻습니다."),
h("div", { class: "row" }, label, btn));
}
// ---- dashboard ----------------------------------------------------------------------------------
/** The header with the start/stop buttons; rebuilt only when the state changes, so a click is not
@@ -543,7 +565,7 @@ async function show(name) {
if (name === "logs") { const l = logsScreen(); screen.replaceChildren(l.view); ticker = setInterval(l.tick, 1000); return; }
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능",
collect: "백그라운드 수집" };
const extra = name === "bot" ? await tokenCard() : null;
const extra = name === "bot" ? await tokenCard() : name === "features" ? await displayDriverCard() : null;
const view = await settingsScreen(name, titles[name], extra);
if (name === "collect") view.append(await collectResults());
if (my === gen) screen.replaceChildren(view);

View File

@@ -271,12 +271,14 @@ LLM과 STT를 함께 켠 측정(M0 `gamemode_stt`, 게임 없음). STT를 먼저
**가상 모니터 방식**: 가상 모니터를 하나 만들고 봇용 크롬을 그 모니터에 둡니다.
- 가상 디스플레이 드라이버는 앱 설치 파일에 들어 있어야 합니다(다른 프로그램 설치를 전제로 하지 않음). 오픈소스 Virtual Display Driver(MIT, SignPath 재단 서명)를 함께 넣고, 앱이 가상 모니터 수를 바꿔 켜고 끕니다(사용자 선택). 설치 때 관리자 권한을 한 번 묻습니다(설치 승인받음). 대상은 x64 Windows 10/11(S 모드 제외)입니다.
- 가상 디스플레이 드라이버는 앱 설치 파일에 들어 있어야 합니다(다른 프로그램 설치를 전제로 하지 않음). 오픈소스 Virtual Display Driver(MIT, SignPath 재단 서명)를 함께 넣고, 앱이 가상 모니터를 켜고 끕니다(사용자 선택). 드라이버의 모니터 수는 0으로 두어도 하나가 생겨서, 모니터 하나를 두고 쓰는 동안에만 바탕 화면에 붙였다가 떼는 방식으로 켜고 끕니다(붙이고 떼는 데는 관리자 권한이 필요 없음). 설치 때 관리자 권한을 한 번 묻고(설치 승인받음), Windows가 "SignPath Foundation의 장치 소프트웨어를 설치할까요?"를 한 번 더 묻습니다. 드라이버 설정은 드라이버(시스템 서비스)가 데이터 폴더를 읽을 수 없어 `%ProgramData%\JarvisBot\display-driver`에 둡니다. 다른 프로그램이 같은 드라이버를 이미 깔아 두었으면 설치하지 않고 그 모니터도 건드리지 않습니다. 대상은 x64 Windows 10/11(S 모드 제외)입니다. 자세한 동작은 `app/src/admin.spec.md` "Virtual monitor".
- 앱이 꺼질 때는 봇용 크롬을 먼저 닫습니다. 가상 모니터가 먼저 사라지면 그 위의 창이 실제 모니터로 옮겨질 수 있기 때문입니다.
- M0 `vdisplay` 단계(Parsec 드라이버로 하는 개념 확인)는 돌리지 않기로 했습니다(사용자 결정). 디스코드 "화면" 탭 공유, 커서, 모니터 해제 때 창 위치는 M6에서 함께 넣을 드라이버로 직접 확인합니다. 드라이버 설치 시험은 게임PC가 아니라 서버의 Windows 가상머신에서 먼저 합니다.
- 마우스가 가상 모니터로 넘어가지 않도록, 실제 모니터와 구석 한 점만 닿게 배치합니다.
- 디스코드에서는 "화면" 탭에서 그 가상 모니터를 고릅니다.
- Alt+Tab에서는 그대로 빠지고, 실제 모니터에도 보이지 않습니다.
- 크롬 창을 처음부터 띄우면 사용자가 하던 게임의 포커스를 가져갈 수 있어서, 창 없이 켠 뒤 앱이 CDP로 창을 뒤에서 엽니다. Windows가 가상 모니터에도 작업 표시줄을 두면 페이지 아래를 가리지 않게 창을 작업 영역에 맞춥니다.
- **서버 Windows 가상머신 확인(2026-11-14)**: 드라이버 설치(SignPath 질문 한 번)와 제거(장치·드라이버 패키지·레지스트리·설정 폴더 남지 않음), 설치 직후 모니터 꺼짐, 봇용 크롬을 켜는 동안만 실제 모니터 오른쪽 아래 구석(1024,768)에 1280x720으로 켜짐, 크롬 창 하나가 그 모니터의 작업 영역을 채우고 봇의 탭도 그 창에 열림, 메모장에 입력하는 동안 포커스를 뺏지 않음, 작업 표시줄에 버튼 없음, 정지 뒤 크롬 프로세스 0개와 모니터 꺼짐. 남은 확인: 디스코드 "화면" 탭에 그 모니터가 나오고 공유되는지, 커서, 게임PC(실제 GPU)에서의 동작.
## 다른 사람의 디스코드 화면공유 보기 (창 분리 방식)

File diff suppressed because one or more lines are too long