- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
viewing"), decides for the asker (only people registered for PC control), never picks, lists
or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each collection target can announce new items at once or in a daily digest
(off by default) in collect.notifyChannel, as titles and links only, with no
LLM; each post is recorded in the activity log. The brain gets a read-only
collectedNews tool (every target's newest items by default, fenced as
untrusted web content), offered with the app's collection file and the
collection feature, and available to scheduled tasks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A personalNotes tool (add, list, delete the asker's own notes) offered only
with a notes backend. The app keeps the notes in config/notes.json and sends
the asker's with every voice, chat and task reply; the reply engine puts them
in a <person_notes> fence as that person's preferences, which never permit an
action or a tool. The worker's schedule bridge becomes a shared app bridge
for schedule_action and note_action. A 사람별 지침 screen edits and deletes
any note; features.notes turns the tool and the notes off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each chat channel action the brain's tool asked for and each schedule
delivery is recorded in activity/<day>.jsonl by kind, target, who asked and
outcome, never its text, kept 30 days. The worker's chat_action carries the
asker. A 한 일 screen lists the newest entries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
permissions::decide gives allow, ask or refuse for an action on the PC: only
people in pcControl.allowedUsers may ask (nobody by default), viewing is
automatic and clicking, typing and running ask first by default, and
payments, passwords, deleting files and administrator rights are never done.
A PC 제어 screen edits the levels and shows the chat channels' send and
delete switches beside them. The PC control tool (M7) will use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brain gets a schedule tool (add, list, delete the asker's own reminders,
timers and tasks) offered only with a schedule backend. The voice worker asks
the app through schedule_action and runs fired tasks as reply via task with
read-only tools, no MCP and a fresh short-term memory. The app keeps the
schedules in config/schedules.json, fires them every 5 s, speaks them when the
person is in Jarvis's voice channel and otherwise posts in the chat channel
they asked in (or schedule.chatChannel for voice). A feature switch and a
예약 screen list and delete everyone's schedules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- update.rs finds the newest app-v<version> release above the running
version (v1 tags, drafts, prereleases, incomplete releases and files on
another server are skipped), downloads Jarvis-Setup-<version>.exe into
cache/updates checked against its .sha256, and hands it over.
- update_install hashes the file again, stops what runs, starts the
installer with /S /RUN as administrator and the window quits; a declined
administrator prompt keeps the app with the services stopped.
- New 업데이트 screen; the app checks once when the main screens open.
- tauri.conf.json no longer sets its own version, so the program file
carries the Cargo.toml version the update compares.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- installer.sh lists every packaged file for the uninstaller, and folders
are removed only when empty, so a chosen folder keeps other files.
- A virtual monitor driver removal that fails stops the uninstall with
everything kept (silent exit code 4); the elevated half exits 2 when
there is no driver of ours to remove, which the uninstaller accepts.
- WebView2 is installed before any file is copied; when it is still
missing the installer asks to retry, or silently stops with code 3
without installing or starting anything.
- The installer holds Global\JarvisBotSetup while it runs; jarvis.exe
started meanwhile shows a message and exits before any window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- while the services run, game mode decides the stage (perf.rs): a game means the game stage at
once (or pause with game.pauseInGame); its end brings the normal stage back after
game.revertSeconds (default 30)
- a change tells the voice worker the LLM switches (llm switching, with a notice), waits until
the brain has written the answers in progress (llm_idle, at most 2 minutes), restarts the LLM in
the stage and tells the worker it is ready or paused
- the game stage runs at 16K context, one request at a time, half the engine's CPU threads and
the game placement (reserve game.reserveGb, default 8, else 2 GiB, else 0 GiB, else pause),
measured once at that context; every placement of a run counts other programs' memory as it
was when the services started
- the voice worker gates every LLM use: on switching new work waits (at most 180 s), the first
spoken request that waits hears the notice once under its own id, on ready all go on, on
paused nothing is answered
- dashboard: "성능 단계: 게임 중 · 게임에 X GB" / 일시정지; game mode screen: game VRAM, revert
wait, pause in game
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- with an NVIDIA card the app sets --expert-vram itself by the M0 kit's formula: what the card
holds before the start, the model without experts, the STT worker's share (2740 MiB), the
engine's margin (768 MiB) and room for programs opened later (game.otherRoomGb, default 1 GB)
are left free; Strata's own placement filled the card and left the STT no room
- the model without experts is measured once per context and card (a start with
expert-vram=0, one short answer, the card's memory and the engine log's staging), kept in
config/llm-probe.json; the real start replaces that server
- placement.rs also works out the game placement (the reserve, else the 2 GiB minimum and the
room a game really gets, else pause) for the switching that comes next
- with the game PC's M0 numbers the normal placement comes out at 1.5 GiB, as M0 measured
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- while the services run the app looks at the foreground window every 2 s (Windows): the
person's notGames and games lists first, never a browser, video player, the desktop, Discord,
a launcher's own window or this app; a program under a Steam, Epic, Riot or Battle.net game
folder, exclusive full screen, or a window covering a monitor without title bar or border whose
process uses at least 10% of the GPU's 3D engine (the M0 rule) is a game
- game mode stays until that game's process ends, also behind other windows; the dashboard shows
"게임 모드: 켜짐 (name, reason)" and the game log tab its changes
- a 게임 모드 settings screen (detection on/off, the two lists)
- checked on the win-ci VM: a program in a Steam game folder turned it on after 1.4 s and off
2.0 s after it ended
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- features.browserShare (봇용 크롬창 화면공유 선택 표시, off by default) and a driver card on the
features screen: the Virtual Display Driver 25.7.23 (MIT, SignPath-signed) ships in
display-driver/ (hash-checked by package.sh); install and uninstall run jarvis.exe elevated
once (--display-driver), settings in %ProgramData%\JarvisBot\display-driver, another
program's copy of the driver is left alone
- the monitor is on the desktop only while the bot browser uses it, at the rightmost real
monitor's bottom-right corner; the stop closes Chrome first, then takes the monitor off
- Chrome starts with no window and the app opens it over CDP in the background, so it never
takes the focus; the window is a tool window (no Alt+Tab, no taskbar button), fills the
monitor's work area and keeps drawing when covered; without a monitor it runs headless with
the reason on the dashboard
- Chrome runs with --do-not-de-elevate on Windows: started by an app running as administrator
it restarted itself as the user and the app lost hold of it
- checked on the win-ci Windows VM from install to uninstall (focus kept while typing, no
taskbar button, no Chrome left, nothing left after uninstall) and on Linux with a real Chrome
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- install step browser: Chrome for Testing 157 (pinned SHA-256) into runtime/chrome
- services start it headless with its own profile when features.browser is on; CDP on a free
loopback port passed to the worker as JARVIS_BROWSER_CDP; a failure leaves the services running
- every Chrome connection goes through the app's loopback egress proxy: one lookup, public
addresses only (403 otherwise, no host logged), QUIC off, WebRTC only through the proxy
- stop: Browser.close, then kill after 5 s; dashboard shows the browser's state
- the worker's runtime environment follows the settings step, not every step being installed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Targets (none by default), interval and keep days on a new 백그라운드 수집 screen with the
results; the worker collects on its own thread through the web search's address check and
writes only to the data folder's collect/ (logs carry target names and counts).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Settings `features` (weather, webSearch, meals) and `mcpServers` (name, command, args, env names,
enabled), edited on the new 기능 screen. The tools of a feature that is off go into the brain's
`disabled_tools`; only enabled MCP servers go into its `mcps`.
- MCP environment values (API keys) live only in the encrypted secrets (`mcp:<server>:<NAME>`),
shown as stored or not, masked in logs, removed with their server, and reach the worker only in
`JARVIS_MCP_ENV`, which the brain takes out of its environment before giving each server its own.
- `enabled_builtin_tools(cfg)` is the one list the router, toolSearchTool and the executor use, so
a turned-off tool is neither offered nor run, also when the model names it.
- The v2 brain never offers v1's `localFiles` (home folder read/write/delete) and `screenshot`:
anyone in the voice or chat channels could ask for them. Screen viewing comes in M7.
- The brain connects MCP servers on its own thread and logs each outcome without debug logging.
- Stored secrets are masked in the log hub as soon as the data folder is found.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-ups to M5.
- Chat channel names and topics (app/src/channel_sync.rs): a channel with
syncDiscord (default on) gets its name and description as its Discord
name and topic when saved; registering fills them from Discord;
Discord's form of the name is written back; a topic over 1,024
characters is refused before saving; a rate-limited edit is pending and
retried once by itself with the latest text; a missing permission fails
with the reason and the app keeps the text. Commands
discord_channel_info and channel_sync_status. Checked on real Discord
with a temporary channel (fill, edit, rate limit, automatic retry).
- voice.tts.speed (0.5-2.0) on the model screen, passed to the worker as
--melo-speed. MeloTTS's Korean model has one voice, so no voice choice.
- Closing the window runs AdminApi::shutdown first: an install in
progress is cancelled and waited for, the services stop, a failed run's
leftovers (Strata) are cleared (Controller::shutdown). If that fails the
window stays open with the reason; closing again quits.
- WebView2 data goes in the data folder (cache/webview), or a temp scratch
folder before one is chosen, removed at the next start: %APPDATA%\JarvisBot
holds only the location file.
- Command output that is not UTF-8 (CP949 on a PC without the UTF-8
option) is decoded in the OEM code page instead of ending the reading.
Verified on the win-ci VM: closing during the install leaves no process,
no --location keeps only location.json under %APPDATA%, the unit tests on
Windows 168/168.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M5 step 4, second part. The services start the LLM first: the installed
Strata with the M0 kit's model and normal placement (models in
models/llm, parallel 1, or 2 when a chat channel answers every message,
pictures off, --restart), wait until /v1/models lists the model, and stop
it last. The first start downloads the model; the dashboard shows
Strata's last line and percentage, and stop also cancels a start (the
server is cleared). llm.url uses a server the app does not start.
On every start the app writes config/brain.json (Strata's /v1, the
listed model id, memory/jarvis.db) and hands it to the worker as
JARVIS_CONFIG_PATH. Loading a config with its own db_path no longer
creates ~/.local/share/jarvis.
service_start/service_restart run in the background. Settings gain
llm.context and llm.port (model screen) and llm.url (file only).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- download: resumed downloads checked against a pinned SHA-256.
- install: uv 0.11.20, the worker/brain venv (Python 3.12, the new
src/voice_worker/requirements-brain.txt), the MeloTTS venv (Python 3.11,
PyTorch cu128), Whisper large-v3-turbo, the Korean MeloTTS voice and
Strata, the M0 kit's recipe; markers per step and version, cancel, retry,
everything (caches, temp, HF and NLTK data) inside the data folder, the uv
cache cleaned and the worker settings pointed at the venvs when done.
- The services give the worker the same environment once installed.
- The wizard's model step shows and drives the install.
Installed from an empty folder on Linux and started the services on it:
worker connected, STT and TTS ready, nothing written outside the folder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- admin_api: the one entry point of the screens (app status, data folder,
PC check, Discord token check and pickers, settings, service control and
the dashboard, logs query, error report and zip export).
- discord_setup: token, Message Content and invite link checks over REST.
- app/ui: first-run wizard, dashboard, five settings screens and the log
screen in plain HTML/CSS/JS (dark theme after DESIGN.md, Korean text).
- jarvis-admin-dev: a loopback, token-guarded development server so the
screens run in a browser against the real commands.
Checked in a browser with the real test bot and voice worker: wizard
steps, start/restart/stop, settings save, chat channels, logs and export.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- control: one Controller starts, stops and restarts the services (stopped,
starting, running, stopping, failed); a failed start leaves nothing
running and keeps its error with secrets masked. The app binary uses it.
- dashboard: built from the core's events alone: the bot's tag, the voice
channel, live captions (screen only, cleared on leaving or stopping), the
last 20 voice answers split into stages with medians (matched by the new
AnswerAsked event), and missed requests without text.
- system: memory in use and per NVIDIA GPU its use and VRAM (nvidia-smi).
- DiscordCore::voice_members, WorkerClient::hello, the conversation's missed
requests shared with the dashboard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- datadir: the data folder layout, the location file (JARVIS_DATA_DIR wins),
free space for the wizard.
- settings_edit: the fields the five settings screens edit; apply writes only
non-defaults, keeps unknown keys and writes nothing unless the app's own
check passes. A memory section reaches the worker's hello.
- secrets: config/secrets.dat, DPAPI for the Windows user; the services take
the bot token from it before JARVIS_DISCORD_TOKEN.
- logs: the log hub behind the log screen (tabs, daily files kept 14 days,
the recent error report, a zip under 10 MB, secrets and token-shaped text
masked); the log macros and the worker's output feed it.
Spec: app/src/admin.spec.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A short two-tone chime, made in code, plays right before each voice answer
starts and at once for a wake word said alone while Jarvis is free; the
question that joins it gets no second one. call.earcon turns it off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- conversation.rs: the loop between the core's events and the rest.
Finals go through call detection and the response policy; answers are
asked of the brain and spoken from the app's queue, an interruption
cancels only that answer, a voice stop silences everything, and missed
requests are kept without text (last 100). Tested with a fake voice side.
- Core: answer(user, text) adopts a playback request the worker speaks
under the same id, cancel(id) stops one request, ask_addressed and
ask_rank pass the conversation's questions; new SpeakingStopped,
Addressed and Ranked events.
- Worker client: reply, addressed and rank messages and their results; a
reply with no answer, refused or not sent ends its speech as failed.
- Voice worker: addressed and rank messages on their own threads, beside
a running reply; a spoken reply cancelled while the brain is answering
is not spoken. judges.py holds the two short LLM judgements with the
heard text fenced as data; eval evals/test_voice_judges.py.
- Specs (conversation, discord core, voice worker), docs/llm_contexts.md
and the roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- calls.rs: wake words anywhere in the utterance with near matches by
edit distance on NFKD-normalised letters (Hangul splits into its
letters), call modes (wakeWord, addressed, everything), designated
people with both switches, and a voice stop (wake word + stop word).
- policy.rs: first, inOrder, latest and important as an event-driven
state machine with the time passed in: ordering by end time, waiting
for an earlier-ended utterance still being transcribed, queue limit,
interruption, importance ranking only on a tie with a timeout, stop
dropping the queue, and a reason for every missed request.
- settings: new call and response sections, validated with the key.
- conversation.spec.md documents the contract; registry and roadmap
updated. Wiring into the Discord core and the brain comes next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user asked for a faster, real app instead of Electron. The app is now a Rust crate (serenity +
songbird); the GUI in M5 will be a Tauri 2 shell on the system WebView2.
- Same behaviour as the Electron core: one voice channel, per-speaker PCM to the voice worker,
speaking_end and speaker_leave, sentence playback with stop and cancel, transcription notice on
join, slash command join/leave/stop (create-or-update), registered chat channels with live events,
history and sends that never ping, worker process with a fresh token and restart backoff.
- songbird 0.6.1 garbles about a third of received frames under DAVE (its decoder keeps the bytes
DAVE decryption leaves at the end of the payload; a 20 ms stepped test tone showed repeats, skips
and reordering, while sending was exact). songbird now only decrypts; the core reorders packets in
a small window (Opus FEC for the last lost frame) and decodes them with libopus itself.
- Live test on real Discord with two bots: five sentences heard (CER 0/0/0/5/12.5%), 0.71-0.90 s
from end of speech to final, speech read back at 3.4%, sound gone 0.08 s after stop. The release
binary uses about 16 MB of memory; a Windows build (MinGW) logs in under Wine.
- Spec moved to app/src/discord_core.spec.md; architecture, decisions, roadmap and pc-features updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>