DuckDuckGo now serves only its bot-check page (game PC and host alike), so
every search fell to Wikipedia full-text search, which picked unrelated
pages (박진영 for songs for a rainy day, 대한민국 for a graphics card
ranking). Live 2026-11-18 answers took 9-15 s and were wrong.
- Exa's free MCP endpoint (no key) is the first provider after the
live-broadcast Chrome; it returns page text, so no fetch is needed.
- Gemini runs only when Exa found nothing and a CLI login (oauth) or key
(apikey) is present; without a login the CLI is no longer started
(about 2 s per search).
- Wikipedia is used only when the page title is named in the query;
otherwise the honest nothing-found envelope.
- Letters of a script the user did not use are dropped from the query
(the model searched '사과와 배 중哪个가 더 단가요').
- DuckDuckGo removed; spec, docs and tests follow the new order.
- weather: any forecast failure (not only requests errors) tries met.no.
Real model (qwen3.6-35b-a3b, game PC), real network: graphics card
ranking, apples vs pears, rainy-day genres and black holes all answered
correctly in 3.7-5.2 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pcControl (v2 M7): windows, read, click, type, keys, run, close through the
app's agent-desktop. Ask-level actions wait for the asker's own next message
and then cover that window's asked kinds for the reply; payments, passwords,
deleting files and admin rights are filtered before asking and never done.
The app refuses during an anti-cheat game and the emergency stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The management app reads the worker's history store (read only) and shows
people's words, Jarvis's answers and the answer traces in time order, with
filters for kind, place, person, channel, day and text, live follow, older
items on demand and expandable traces (records given, tools offered, plan,
model thinking, tool calls and results).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
viewing"), decides for the asker (only people registered for PC control), never picks, lists
or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each collection target can announce new items at once or in a daily digest
(off by default) in collect.notifyChannel, as titles and links only, with no
LLM; each post is recorded in the activity log. The brain gets a read-only
collectedNews tool (every target's newest items by default, fenced as
untrusted web content), offered with the app's collection file and the
collection feature, and available to scheduled tasks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A personalNotes tool (add, list, delete the asker's own notes) offered only
with a notes backend. The app keeps the notes in config/notes.json and sends
the asker's with every voice, chat and task reply; the reply engine puts them
in a <person_notes> fence as that person's preferences, which never permit an
action or a tool. The worker's schedule bridge becomes a shared app bridge
for schedule_action and note_action. A 사람별 지침 screen edits and deletes
any note; features.notes turns the tool and the notes off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brain gets a schedule tool (add, list, delete the asker's own reminders,
timers and tasks) offered only with a schedule backend. The voice worker asks
the app through schedule_action and runs fired tasks as reply via task with
read-only tools, no MCP and a fresh short-term memory. The app keeps the
schedules in config/schedules.json, fires them every 5 s, speaks them when the
person is in Jarvis's voice channel and otherwise posts in the chat channel
they asked in (or schedule.chatChannel for voice). A feature switch and a
예약 screen list and delete everyone's schedules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
open, read, click (label or [n]), type (optionally Enter) and back; page content comes back fenced
as untrusted data; only http(s) addresses; no LLM call. Offered only when the app started the bot
browser (JARVIS_BROWSER_CDP) and the browser feature is on; meanwhile fetchWebPage, which has no
address guard, is not offered. Eval evals/test_bot_browser_tool.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M5 step 5, first part. app/shell is a Tauri 2 crate (jarvis.exe): one
window over app/ui whose invoke("admin") goes to AdminApi::call. Its
WebView2 data sits next to the location file; closing the window stops
the services and waits for a cancelled start (AdminApi::shutdown).
app/shell/package.sh cross-builds it on Linux (x86_64-pc-windows-gnu)
and lays out Jarvis/ with WebView2Loader.dll and python-src/
(voice_worker, jarvis). The worker's default sources are python-src next
to the executable when present, else the repository's src/.
The bot screen now shows the masked token right after saving.
Checked on the win-ci VM: the window and wizard, PC check, uv and worker
venv installed, a cancel leaving no uv/Python process, the token stored
DPAPI-encrypted, a start failing with its reason, logs, error copy, zip
export, closing the window ending the process, nothing under %APPDATA% or
%LOCALAPPDATA%; the core's unit tests on Windows (156/157, the one
Linux-path assumption fixed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- datadir: the data folder layout, the location file (JARVIS_DATA_DIR wins),
free space for the wizard.
- settings_edit: the fields the five settings screens edit; apply writes only
non-defaults, keeps unknown keys and writes nothing unless the app's own
check passes. A memory section reaches the worker's hello.
- secrets: config/secrets.dat, DPAPI for the Windows user; the services take
the bot token from it before JARVIS_DISCORD_TOKEN.
- logs: the log hub behind the log screen (tabs, daily files kept 14 days,
the recent error report, a zip under 10 MB, secrets and token-shaped text
masked); the log macros and the worker's output feed it.
Spec: app/src/admin.spec.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- calls.rs: wake words anywhere in the utterance with near matches by
edit distance on NFKD-normalised letters (Hangul splits into its
letters), call modes (wakeWord, addressed, everything), designated
people with both switches, and a voice stop (wake word + stop word).
- policy.rs: first, inOrder, latest and important as an event-driven
state machine with the time passed in: ordering by end time, waiting
for an earlier-ended utterance still being transcribed, queue limit,
interruption, importance ranking only on a tie with a timeout, stop
dropping the queue, and a reason for every missed request.
- settings: new call and response sections, validated with the key.
- conversation.spec.md documents the contract; registry and roadmap
updated. Wiring into the Discord core and the brain comes next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Tool (chat_channels.spec.md): list, read, search, send, edit, react,
delete, pin, unpin. Offered only when the v2 brain gives the turn a chat
backend, and then on every turn. Channel by ID, name, part of a name or a
name inside the text; a message ID finds its channel; planner forms of the
action ("send_message") work. Read and search results are fenced as
untrusted data. delete only reports until called again with confirm=true
in a later turn of the same person.
- Worker: chat_channels from the app, ChatBridge reads/searches the history
store and sends chat_action, waiting for chat_action_result (15 s timeout,
failed on session close); per-person reply turns and pending deletes.
- App: channel switches edit/react (on) and delete/pin (off); chat_channels
sent on worker ready and after login; chat_action runs send, edit (own
messages only), react, delete, pin, unpin and answers with ids or the error.
- Engine: chatChannels left out of the catalogue without a backend; the
"router returned everything" check now tolerates that.
- live_discord: edit own, refuse editing the speaker's, react, pin (Discord
refuses in a voice channel's chat, reported as an error), delete, refuse an
unregistered channel; passed in 109.4 s.
- Eval (qwen2.5:7b, 3 runs): 10 of 12; never obeyed the "delete everything"
message and never deleted in the same turn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/jarvis/memory/history.py keeps voice transcripts, Jarvis's answers,
registered chat channel messages (with edits and deletes) and voice channel
presence in one SQLite file. Every read is scoped to the asker: own records
always, chat channels and others' words (only while the asker was in the
same voice channel) behind settings, plus an optional since and off.
- Search is language-agnostic: Unicode words, prefixes keeping 60% of the
word, rarer words weigh more, newer wins ties. 100k records search in
24-61 ms here. context() returns the recent window and the matches cut to a
token budget, matches at most half so chatter cannot push out the record
the question is about.
- run_reply_engine(history=...) puts that context into the system prompt
under "Information shared in previous conversations", fenced as data.
evals/test_history_context_recall.py: with the block qwen2.5:7b answers the
stored day in Korean and English (3 of 3 runs), without it the day is unknown.
- Spec history.spec.md, registry entry in CLAUDE.md, reply.spec.md,
docs/llm_contexts.md, docs/v2/memory.md and roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user asked for a faster, real app instead of Electron. The app is now a Rust crate (serenity +
songbird); the GUI in M5 will be a Tauri 2 shell on the system WebView2.
- Same behaviour as the Electron core: one voice channel, per-speaker PCM to the voice worker,
speaking_end and speaker_leave, sentence playback with stop and cancel, transcription notice on
join, slash command join/leave/stop (create-or-update), registered chat channels with live events,
history and sends that never ping, worker process with a fresh token and restart backoff.
- songbird 0.6.1 garbles about a third of received frames under DAVE (its decoder keeps the bytes
DAVE decryption leaves at the end of the payload; a 20 ms stepped test tone showed repeats, skips
and reordering, while sending was exact). songbird now only decrypts; the core reorders packets in
a small window (Opus FEC for the last lost frame) and decodes them with libopus itself.
- Live test on real Discord with two bots: five sentences heard (CER 0/0/0/5/12.5%), 0.71-0.90 s
from end of speech to final, speech read back at 3.4%, sound gone 0.08 s after stop. The release
binary uses about 16 MB of memory; a Windows build (MinGW) logs in under Wine.
- Spec moved to app/src/discord_core.spec.md; architecture, decisions, roadmap and pc-features updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- app/ (Electron 45 + TypeScript): the core joins one voice channel, decodes each speaker's Opus to
48 kHz stereo and streams it to the voice worker, sends speaking_end and speaker_leave, and emits
speech_start/partial/final with the speaker's display name.
- Sentence playback from the worker's TTS, first come first served, with stop that silences the clip
at once and cancels unfinished requests; slash command join/leave/stop (create-or-update by name).
- Transcription notice posted on every join; other bots ignored by default; at most maxSpeakers
people transcribed at once.
- Registered chat channels: live message events, history reads, sends split at 2,000 characters with
mentions disabled; Message Content intent checked before login.
- The app starts the Python voice worker with a fresh token in the environment, restarts it with
backoff, and the worker client reconnects and fails in-flight TTS on a lost connection.
- Spec in app/src/core/discord_core.spec.md; Vitest unit tests, a protocol test against the real
Python worker server with fake engines, and an opt-in live test with two bots on real Discord.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One shared faster-whisper model for every speaker, batched; finals go before partials and
partial batches are capped at two so a final never waits behind a ten-speaker pass.
- Per-speaker 48k stereo to 16k mono conversion, streaming Silero VAD, utterance ends on VAD
silence, Discord speaking_end, missing audio, or the 28 s Whisper window.
- MeloTTS in its own venv through a child process; sentences stream out one chunk each, with
incremental text, cancel and failure reporting.
- Loopback-only server with a token from the environment; logs carry no transcript text.
- Spec in src/voice_worker/voice_worker.spec.md; unit tests with fake engines and an opt-in
real-GPU test that streams ten speakers at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- run-chrome.sh: render web content (YouTube/Google/Naver) in Korean. --lang
only sets Chrome's UI; the Accept-Language sent to sites comes from the
profile's intl.accept_languages, which a persisted user-data-dir kept at
en-US. Seed the profile to ko-KR and add --accept-lang=ko-KR,ko.
- agents/llm.md: runtime instructions for the reply LLM (loaded by the agents
feature) — name "자비스", concise 1-2 sentence TTS replies (no emojis/markdown),
nuance-based intent, and YouTube voice controls (open/search/play Nth/pause/
back) via the on-screen browser tool.
- CLAUDE.md: drop the "use emojis in CLI output" rule — this assistant replies by
Discord voice, not CLI, so output should be plain.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the two info modes in the Python brain:
- config.py: read STREAM_BROWSER / GEMINI_API_KEY / GEMINI_MODEL from env into
Settings (stream_browser, gemini_api_key, gemini_model). Verified load_settings
reads both modes.
- realtime_search.py: two fail-open backends returning the same fenced
UNTRUSTED-WEB-EXTRACT envelope: browser_search() shells the Node CDP helper to
drive the on-screen Chrome (visible on the broadcast); gemini_search() calls
the Gemini REST API with google_search grounding.
- web_search.run(): routes by mode before the DDG cascade (true->browser,
false->Gemini), falling through to DDG/Brave/Wikipedia on any miss.
- browse_and_play tool: plays a YouTube video on the shared screen (true mode
only); registered in the tool registry.
- specs + docs/llm_contexts.md updated (new Gemini LLM context); CLAUDE.md spec
registry updated.
Verified live against the running Chrome: true-mode webSearch returned real
Google results for "오늘 서울 날씨", browseAndPlay played the IU 밤편지 MV, and
false-mode degrades gracefully on a bad/absent key. A valid GEMINI_API_KEY is
still needed to confirm the real Gemini grounding output.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>