- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
viewing"), decides for the asker (only people registered for PC control), never picks, lists
or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the shared screen a tab brought to the front makes Chrome bring its window forward and take the
keyboard focus from the game or program the user is in (seen on the game PC). The botBrowser tool
now opens each person's page in a window of its own, behind and at the place and size of the bot
browser's window (Target.createTarget with newWindow and background), and closes it again if the
connection to it fails. While the browser runs on the virtual monitor, the app looks for new Chrome
windows every 200 ms and settles each like the first (tool window, fitted to the work area, without
activating it), the newest in front of the older ones, so the shared screen shows the page the bot
opened last.
Checked on the game PC: the bot's window loading a public page comes in front of the first, both
out of the taskbar and Alt+Tab, the focus staying in Discord also with the user idle for hours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chrome 157 no longer reads --disable-features=Translate, so on the virtual monitor every page in
another language got a translate bubble over it on the shared screen (seen on the game PC). The
profile's preferences now turn translation off before each start, keeping the other preferences.
The live screen test also checks that opening the window leaves the focus where it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The app writes the reason in UTF-8; FileRead took it for the system's ANSI
code page, so a declined driver install showed mojibake in the message box
and in the details. Read the file's bytes and convert them from UTF-8, for the
install and the uninstall message alike.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The driver component is on by default, so installing Jarvis installs the
driver too (Windows asks the SignPath question once). A silent install, which
is how updates run, never touches the driver: no question during an update and
no driver brought back after the person removed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turning a daily target on only marks its current items as seen, so that
day's digest still goes out at its hour. Each round saves its state before
posting and posts nothing when the save fails, so the same items are not
posted again every minute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each collection target can announce new items at once or in a daily digest
(off by default) in collect.notifyChannel, as titles and links only, with no
LLM; each post is recorded in the activity log. The brain gets a read-only
collectedNews tool (every target's newest items by default, fenced as
untrusted web content), offered with the app's collection file and the
collection feature, and available to scheduled tasks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A personalNotes tool (add, list, delete the asker's own notes) offered only
with a notes backend. The app keeps the notes in config/notes.json and sends
the asker's with every voice, chat and task reply; the reply engine puts them
in a <person_notes> fence as that person's preferences, which never permit an
action or a tool. The worker's schedule bridge becomes a shared app bridge
for schedule_action and note_action. A 사람별 지침 screen edits and deletes
any note; features.notes turns the tool and the notes off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chat action's channel, message and asker are recorded only when they are
ids (else '(잘못된 ID)'), an unknown action as chat.other, and a failure as a
fixed note, so text the model put in those fields never reaches the log. Old
day files are also deleted on the first record of each new day.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each chat channel action the brain's tool asked for and each schedule
delivery is recorded in activity/<day>.jsonl by kind, target, who asked and
outcome, never its text, kept 30 days. The worker's chat_action carries the
asker. A 한 일 screen lists the newest entries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
permissions::decide gives allow, ask or refuse for an action on the PC: only
people in pcControl.allowedUsers may ask (nobody by default), viewing is
automatic and clicking, typing and running ask first by default, and
payments, passwords, deleting files and administrator rights are never done.
A PC 제어 screen edits the levels and shows the chat channels' send and
delete switches beside them. The PC control tool (M7) will use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A repeating schedule the app was off for is judged by its latest time up to
now, so today's reminder is not skipped after a night off.
- A spoken reminder counts as delivered only when its speech ends played to
the end; speech that failed or was stopped goes to the chat channel.
- A schedule being delivered is marked firing; one still marked at the next
start gets a failed outcome instead of vanishing without a reason.
- Deliveries run in the loop's JoinSet, so stopping the services stops them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brain gets a schedule tool (add, list, delete the asker's own reminders,
timers and tasks) offered only with a schedule backend. The voice worker asks
the app through schedule_action and runs fired tasks as reply via task with
read-only tools, no MCP and a fresh short-term memory. The app keeps the
schedules in config/schedules.json, fires them every 5 s, speaks them when the
person is in Jarvis's voice channel and otherwise posts in the chat channel
they asked in (or schedule.chatChannel for voice). A feature switch and a
예약 screen list and delete everyone's schedules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All five proactive proposals in the recommended order with their defaults;
PC control uses the three-level permissions and only IDs registered in the
app may ask; the virtual monitor check on the game PC, GPU time on request,
and releases on the public Gitea repository are approved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- update.rs finds the newest app-v<version> release above the running
version (v1 tags, drafts, prereleases, incomplete releases and files on
another server are skipped), downloads Jarvis-Setup-<version>.exe into
cache/updates checked against its .sha256, and hands it over.
- update_install hashes the file again, stops what runs, starts the
installer with /S /RUN as administrator and the window quits; a declined
administrator prompt keeps the app with the services stopped.
- New 업데이트 screen; the app checks once when the main screens open.
- tauri.conf.json no longer sets its own version, so the program file
carries the Cargo.toml version the update compares.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The lock is closed in a last hidden section once every change is done,
so /S /RUN and the finish page's "Jarvis 실행" start the app instead of
its "installing" message.
- An installer or uninstaller that finds the lock taken (or cannot make
it) stops at once, silently with exit code 5, so two never run together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- installer.sh lists every packaged file for the uninstaller, and folders
are removed only when empty, so a chosen folder keeps other files.
- A virtual monitor driver removal that fails stops the uninstall with
everything kept (silent exit code 4); the elevated half exits 2 when
there is no driver of ours to remove, which the uninstaller accepts.
- WebView2 is installed before any file is copied; when it is still
missing the installer asks to retry, or silently stops with code 3
without installing or starting anything.
- The installer holds Global\JarvisBotSetup while it runs; jarvis.exe
started meanwhile shows a message and exits before any window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- app/shell/installer.nsi with installer.sh: package.sh's folder into Jarvis-Setup-<version>.exe
(NSIS, about 10 MB) and its .sha256; models and runtimes still come with the first-run wizard
- Program Files with one administrator prompt, Korean pages, components (start menu and desktop
shortcuts, the virtual monitor's driver off by default, installed through the app's elevated
half), the WebView2 runtime only when missing, a wait for a running Jarvis, /S and /S /RUN for
updates, the app started through Explorer so it never runs as administrator
- uninstall removes this app's driver, only the files it installed, the shortcuts and its
registry keys; the data folder stays
- checked on the win-ci VM: silent install, start, update while running and start again,
uninstall; and an install with the driver from the pages, whose uninstall leaves no device,
driver package or registry key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the app's ready message says whether the game stage runs (llm ready game=true); the worker then
holds partial passes while a reply is being written and runs them again, the newest audio per
speaker, when no answer is; finals never wait; without a game nothing changes
- M0 gamemode_tune decision: the game placement answered 1.7 times faster with partials held
(64.9 against 37.5 tok/s with a partial pass every 2 s)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- a normal placement keeps its reading only when no game runs: the perf loop passes whether a
game runs to every switch, so the way back from a failed game stage while the game runs keeps
the reading from before the game
- game detection also looks at every running program once as the services start: a program in
a launcher's game folder or the person's games list counts, also behind other windows
- checked on the win-ci VM: a minimised program in a Steam game folder is found at the start
while nothing is in front
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the game stage reads the card again: other programs as at the last normal placement made with
no game running, and a reserve of at least what grew since (the game and programs opened after
that reading); with a game already running at the start (game detection now looks once before
the LLM starts) it counts what the card holds now and leaves only the room for other programs,
so it no longer pauses for counting the game twice
- the retry count starts over whenever another stage is wanted, also when the game ends with the
normal stage already running, so the next game tries again
- the wait before the next try counts from when the failed switch and its way back are over
- the worker client keeps, sends and readies a connection under one lock (llm, then state), so an
LLM state set while a worker connects is not lost
- a game watch dropped by a start that fails part way stops its thread
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the normal stage reads other programs' memory afresh (the card after the LLM stopped, less the
STT worker's model) and keeps it; the game stage counts from the latest normal reading, so the
game is not counted against its own reserve
- a restart that fails brings back the stage that ran before and tells the brain ready; the
failed stage is tried again after 60 s, three times at most; when nothing comes back the brain
is told paused so requests fail at once
- the worker client keeps the latest llm state and sends it first on every connection
- switches are numbered: llm switching carries an id, llm_idle names it, and a late idle of an
earlier switch no longer lets a new one through
- a request that waited 180 s for a switch fails without calling the brain
- the dashboard shows the stage that runs also on a fallback placement without experts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- while the services run, game mode decides the stage (perf.rs): a game means the game stage at
once (or pause with game.pauseInGame); its end brings the normal stage back after
game.revertSeconds (default 30)
- a change tells the voice worker the LLM switches (llm switching, with a notice), waits until
the brain has written the answers in progress (llm_idle, at most 2 minutes), restarts the LLM in
the stage and tells the worker it is ready or paused
- the game stage runs at 16K context, one request at a time, half the engine's CPU threads and
the game placement (reserve game.reserveGb, default 8, else 2 GiB, else 0 GiB, else pause),
measured once at that context; every placement of a run counts other programs' memory as it
was when the services started
- the voice worker gates every LLM use: on switching new work waits (at most 180 s), the first
spoken request that waits hears the notice once under its own id, on ready all go on, on
paused nothing is answered
- dashboard: "성능 단계: 게임 중 · 게임에 X GB" / 일시정지; game mode screen: game VRAM, revert
wait, pause in game
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- strata launch --stop now always runs before the card is read and the start's own port must
refuse connections; a left server that answers, still loads (503) or stays silent was skipped
before and its memory counted as other programs'
- a stop that is not confirmed reads and keeps nothing: the model starts with expert-vram=0
- the stop confirms on the port it is given (stop_server_on), not on one remembered from an
earlier start
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- a measurement whose short answer fails, or whose second card reading fails or shows nothing
more held, is not kept: the model starts with expert-vram=0 and the next start measures again
- measurements are kept per context and requests at once (<context>/<parallel>) and per card
(its NVIDIA UUID), not by the card's size
- a server on this app's port left from a run that ended badly is stopped before the card is
read, so its memory is not counted as other programs'
- the game placement falls back to 0 GiB of experts before pausing when 2 GiB leaves a game
nothing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- with an NVIDIA card the app sets --expert-vram itself by the M0 kit's formula: what the card
holds before the start, the model without experts, the STT worker's share (2740 MiB), the
engine's margin (768 MiB) and room for programs opened later (game.otherRoomGb, default 1 GB)
are left free; Strata's own placement filled the card and left the STT no room
- the model without experts is measured once per context and card (a start with
expert-vram=0, one short answer, the card's memory and the engine log's staging), kept in
config/llm-probe.json; the real start replaces that server
- placement.rs also works out the game placement (the reserve, else the 2 GiB minimum and the
room a game really gets, else pause) for the switching that comes next
- with the game PC's M0 numbers the normal placement comes out at 1.5 GiB, as M0 measured
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Battle.net installs anywhere: its games come from the machine's uninstall entries (removed by
its Blizzard Uninstaller or published by Blizzard Entertainment, with an install folder, not
Battle.net itself), read when the watch starts; a program inside one is a game named after
the entry
- launcher folders are matched part by part with ASCII case ignored; the name was cut by byte
positions of a lower-cased copy, which panicked the detection thread on a path such as
D:\İ\steamapps\common\하데스\game.exe
- checked on the win-ci VM: a Steam game under a non-ASCII path and a Battle.net game (test
uninstall entry) turn game mode on with their names and off when they end
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- while the services run the app looks at the foreground window every 2 s (Windows): the
person's notGames and games lists first, never a browser, video player, the desktop, Discord,
a launcher's own window or this app; a program under a Steam, Epic, Riot or Battle.net game
folder, exclusive full screen, or a window covering a monitor without title bar or border whose
process uses at least 10% of the GPU's 3D engine (the M0 rule) is a game
- game mode stays until that game's process ends, also behind other windows; the dashboard shows
"게임 모드: 켜짐 (name, reason)" and the game log tab its changes
- a 게임 모드 settings screen (detection on/off, the two lists)
- checked on the win-ci VM: a program in a Steam game folder turned it on after 1.4 s and off
2.0 s after it ended
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- a failed install forgets the driver (registry value, settings folder) only when its clean-up
left no device and no package, and never when it was ours before; otherwise it stays ours and
the error gives both the install's reason and that something is left to remove
- the driver status separates a usable monitor (installed) from what this app left without one
(leftover) and names the button (action): uninstall for both, so a removal whose package step
failed can be retried from the features screen
- pure rules after_failed_install and driver_status with unit tests; checked again on the win-ci
VM: a declined install leaves nothing, then install and uninstall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- installing or removing the driver is refused unless the services are stopped (or failed), and
a service start or restart (and a second driver task) is refused while one runs, so the
monitor never vanishes under the bot browser's window
- a failed or declined install undoes what that attempt made (device, and the registry value and
settings folder when the driver was not ours before); a declined install's
0xE0000243 reads as cancelled; installed means ours with its monitor there
- devices are found among all root devices whatever their class: a device whose driver was
declined drops out of the display class and was left behind
- uninstall keeps the registry value (what makes it ours) until the device and the driver
package are gone (a package not in the store, 1168, counts as gone), so a failed removal can
be retried
- checked on the win-ci VM: a declined install leaves nothing, the next install gives one working
device, the uninstall leaves nothing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- features.browserShare (봇용 크롬창 화면공유 선택 표시, off by default) and a driver card on the
features screen: the Virtual Display Driver 25.7.23 (MIT, SignPath-signed) ships in
display-driver/ (hash-checked by package.sh); install and uninstall run jarvis.exe elevated
once (--display-driver), settings in %ProgramData%\JarvisBot\display-driver, another
program's copy of the driver is left alone
- the monitor is on the desktop only while the bot browser uses it, at the rightmost real
monitor's bottom-right corner; the stop closes Chrome first, then takes the monitor off
- Chrome starts with no window and the app opens it over CDP in the background, so it never
takes the focus; the window is a tool window (no Alt+Tab, no taskbar button), fills the
monitor's work area and keeps drawing when covered; without a monitor it runs headless with
the reason on the dashboard
- Chrome runs with --do-not-de-elevate on Windows: started by an app running as administrator
it restarted itself as the user and the app lost hold of it
- checked on the win-ci Windows VM from install to uninstall (focus kept while typing, no
taskbar button, no Chrome left, nothing left after uninstall) and on Linux with a real Chrome
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- each asker has a tab and connection of their own (at most 5, least recently used closes first)
- links, forms and window.open that would open a new tab load in the same tab
- one 15 s deadline covers connecting, the action and the page; events do not extend it
- a broken connection retries only open and read; click, type and back may have run and are not repeated
- <<< and >>> in page content are replaced so it cannot close the fence
- the description says how to search on a site (open, then type with submit=true); eval case 0/3 -> 3/3 on qwen3:8b
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
open, read, click (label or [n]), type (optionally Enter) and back; page content comes back fenced
as untrusted data; only http(s) addresses; no LLM call. Offered only when the app started the bot
browser (JARVIS_BROWSER_CDP) and the browser feature is on; meanwhile fetchWebPage, which has no
address guard, is not offered. Eval evals/test_bot_browser_tool.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- IPv6 counts as public only in 2000::/3 (without 2001::/23, documentation and 6to4), so site
local fec0::/10 and IPv4-compatible addresses are refused; the collector uses the same rule
- every proxy connection is a task of one set: a stop ends the open CONNECT tunnels and waits
until their sockets are gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- install step browser: Chrome for Testing 157 (pinned SHA-256) into runtime/chrome
- services start it headless with its own profile when features.browser is on; CDP on a free
loopback port passed to the worker as JARVIS_BROWSER_CDP; a failure leaves the services running
- every Chrome connection goes through the app's loopback egress proxy: one lookup, public
addresses only (403 otherwise, no host logged), QUIC off, WebRTC only through the proxy
- stop: Browser.close, then kill after 5 s; dashboard shows the browser's state
- the worker's runtime environment follows the settings step, not every step being installed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The request line is ASCII: other characters and spaces in the path and query go as UTF-8 %HH,
while existing %HH and separators stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The name lookup runs on a daemon thread waited for only until the deadline, and a watchdog
shuts the connection down at the deadline, so a server that sends its status line or headers
byte by byte within the per-read limit is cut off too. A response cut short is a failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the fetcher resolves once, requires every address to be public and connects only to those
(TLS still checks the name), so DNS rebinding cannot reach the local network; redirects too
- 15 s for the whole fetch also against servers that send slowly; no URL in any log line
- a site whose title alone changed is kept again
- target names that differ only in case are refused (one Windows folder)
- the worker always gets the collection file, so results older than the keep days go also
with no targets or the feature off
- the results screen lists every target with results, not only those in the newest 100
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Targets (none by default), interval and keep days on a new 백그라운드 수집 screen with the
results; the worker collects on its own thread through the web search's address check and
writes only to the data folder's collect/ (logs carry target names and counts).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers the admin paths (`mcp_env_set`, `settings_apply` removing a server): after the services
started with the old values, a late worker line carrying them is masked on the log screen and in
the error report. Fails with the in-use values dropped from the masks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each worker run's log sink keeps the values masked when it started (its secrets in use), so a
line still being logged after a stop gave up waiting, and after the next start replaced the values
in use, is still masked. The debug echo on stderr is masked the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stopping (and restarting) the voice worker now waits, at most 5 s, for its output readers to log
what the exited worker wrote, so its last lines are masked with the secrets that run was started
with before the next start replaces them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The log hub masks the stored secrets and the values the services were started with (bot token,
MCP environment values), so a late worker or MCP line carrying a key replaced or removed while the
services run stays masked on the screen, in copies, files and exports; the next start replaces
the in-use values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Discovery runs outside the MCP tool cache's lock (one at a time), so a turn reading the cached
tools never waits for a slow server; a new conversation refreshes in the background.
- The log hub masks secrets before a line is kept or written, so a key replaced or removed later
stays hidden on the screen, in copies, files and exports, also after a restart.
- The worker takes JARVIS_MCP_ENV out of its environment at start, before MeloTTS runs, and hands
it to the brain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Settings `features` (weather, webSearch, meals) and `mcpServers` (name, command, args, env names,
enabled), edited on the new 기능 screen. The tools of a feature that is off go into the brain's
`disabled_tools`; only enabled MCP servers go into its `mcps`.
- MCP environment values (API keys) live only in the encrypted secrets (`mcp:<server>:<NAME>`),
shown as stored or not, masked in logs, removed with their server, and reach the worker only in
`JARVIS_MCP_ENV`, which the brain takes out of its environment before giving each server its own.
- `enabled_builtin_tools(cfg)` is the one list the router, toolSearchTool and the executor use, so
a turned-off tool is neither offered nor run, also when the model names it.
- The v2 brain never offers v1's `localFiles` (home folder read/write/delete) and `screenshot`:
anyone in the voice or chat channels could ask for them. Screen viewing comes in M7.
- The brain connects MCP servers on its own thread and logs each outcome without debug logging.
- Stored secrets are masked in the log hub as soon as the data folder is found.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A rate-limited edit of a newly registered channel is retried with the
same "never clear Discord with an empty field" rule (it was retried as
an existing channel and cleared the topic).
- Channel edits go one at a time, and Discord's form of the name is
written back only while the settings still hold the name that was sent,
so a slow answer to an earlier save never overwrites a later one.
- The chat channel screen drops a Discord name/topic lookup that answers
after the ID was changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>