- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
viewing"), decides for the asker (only people registered for PC control), never picks, lists
or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A cancelled command ends with every process it started (process group on
Unix, taskkill /T on Windows).
- Waiting for a response or the next chunk gives way to a cancel, so a
silent server cannot hold the install.
- Pointing the settings at the runtime is the last install step: a failure
shows as failed and can be retried; done means written.
- Strata's installer gets its folder as the working folder, not inside the
script text.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- download: resumed downloads checked against a pinned SHA-256.
- install: uv 0.11.20, the worker/brain venv (Python 3.12, the new
src/voice_worker/requirements-brain.txt), the MeloTTS venv (Python 3.11,
PyTorch cu128), Whisper large-v3-turbo, the Korean MeloTTS voice and
Strata, the M0 kit's recipe; markers per step and version, cancel, retry,
everything (caches, temp, HF and NLTK data) inside the data folder, the uv
cache cleaned and the worker settings pointed at the venvs when done.
- The services give the worker the same environment once installed.
- The wizard's model step shows and drives the install.
Installed from an empty folder on Linux and started the services on it:
worker connected, STT and TTS ready, nothing written outside the folder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- datadir: the data folder layout, the location file (JARVIS_DATA_DIR wins),
free space for the wizard.
- settings_edit: the fields the five settings screens edit; apply writes only
non-defaults, keeps unknown keys and writes nothing unless the app's own
check passes. A memory section reaches the worker's hello.
- secrets: config/secrets.dat, DPAPI for the Windows user; the services take
the bot token from it before JARVIS_DISCORD_TOKEN.
- logs: the log hub behind the log screen (tabs, daily files kept 14 days,
the recent error report, a zip under 10 MB, secrets and token-shaped text
masked); the log macros and the worker's output feed it.
Spec: app/src/admin.spec.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- calls.rs: wake words anywhere in the utterance with near matches by
edit distance on NFKD-normalised letters (Hangul splits into its
letters), call modes (wakeWord, addressed, everything), designated
people with both switches, and a voice stop (wake word + stop word).
- policy.rs: first, inOrder, latest and important as an event-driven
state machine with the time passed in: ordering by end time, waiting
for an earlier-ended utterance still being transcribed, queue limit,
interruption, importance ranking only on a tie with a timeout, stop
dropping the queue, and a reason for every missed request.
- settings: new call and response sections, validated with the key.
- conversation.spec.md documents the contract; registry and roadmap
updated. Wiring into the Discord core and the brain comes next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user asked for a faster, real app instead of Electron. The app is now a Rust crate (serenity +
songbird); the GUI in M5 will be a Tauri 2 shell on the system WebView2.
- Same behaviour as the Electron core: one voice channel, per-speaker PCM to the voice worker,
speaking_end and speaker_leave, sentence playback with stop and cancel, transcription notice on
join, slash command join/leave/stop (create-or-update), registered chat channels with live events,
history and sends that never ping, worker process with a fresh token and restart backoff.
- songbird 0.6.1 garbles about a third of received frames under DAVE (its decoder keeps the bytes
DAVE decryption leaves at the end of the payload; a 20 ms stepped test tone showed repeats, skips
and reordering, while sending was exact). songbird now only decrypts; the core reorders packets in
a small window (Opus FEC for the last lost frame) and decodes them with libopus itself.
- Live test on real Discord with two bots: five sentences heard (CER 0/0/0/5/12.5%), 0.71-0.90 s
from end of speech to final, speech read back at 3.4%, sound gone 0.08 s after stop. The release
binary uses about 16 MB of memory; a Windows build (MinGW) logs in under Wine.
- Spec moved to app/src/discord_core.spec.md; architecture, decisions, roadmap and pc-features updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>