Commit Graph

19 Commits

Author SHA1 Message Date
javis-bot
53bccb620d feat: screen viewing, Jarvis sees the whole screen, a monitor or a window for registered people
- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
  title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
  viewing"), decides for the asker (only people registered for PC control), never picks, lists
  or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
  minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
  OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
  not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
  game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 17:02:59 +09:00
javis-bot
324fd2ea0a feat: news notifications for the background collection and a collectedNews tool
Each collection target can announce new items at once or in a daily digest
(off by default) in collect.notifyChannel, as titles and links only, with no
LLM; each post is recorded in the activity log. The brain gets a read-only
collectedNews tool (every target's newest items by default, fenced as
untrusted web content), offered with the app's collection file and the
collection feature, and available to scheduled tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:22:54 +09:00
javis-bot
a5bd25b86f feat: personal notes, each person's standing instructions for how Jarvis answers them
A personalNotes tool (add, list, delete the asker's own notes) offered only
with a notes backend. The app keeps the notes in config/notes.json and sends
the asker's with every voice, chat and task reply; the reply engine puts them
in a <person_notes> fence as that person's preferences, which never permit an
action or a tool. The worker's schedule bridge becomes a shared app bridge
for schedule_action and note_action. A 사람별 지침 screen edits and deletes
any note; features.notes turns the tool and the notes off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:10:53 +09:00
javis-bot
ea2d3ba958 feat: reminders and scheduled tasks (schedule tool, app store and delivery)
The brain gets a schedule tool (add, list, delete the asker's own reminders,
timers and tasks) offered only with a schedule backend. The voice worker asks
the app through schedule_action and runs fired tasks as reply via task with
read-only tools, no MCP and a fresh short-term memory. The app keeps the
schedules in config/schedules.json, fires them every 5 s, speaks them when the
person is in Jarvis's voice channel and otherwise posts in the chat channel
they asked in (or schedule.chatChannel for voice). A feature switch and a
예약 screen list and delete everyone's schedules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 16:27:42 +09:00
javis-bot
56212c06dc fix(brain): botBrowser keeps a tab per person, one deadline, and never repeats an action
- each asker has a tab and connection of their own (at most 5, least recently used closes first)
- links, forms and window.open that would open a new tab load in the same tab
- one 15 s deadline covers connecting, the action and the page; events do not extend it
- a broken connection retries only open and read; click, type and back may have run and are not repeated
- <<< and >>> in page content are replaced so it cannot close the fence
- the description says how to search on a site (open, then type with submit=true); eval case 0/3 -> 3/3 on qwen3:8b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:33:00 +09:00
javis-bot
077db5031c feat(brain): botBrowser tool drives the bot's own Chrome over CDP
open, read, click (label or [n]), type (optionally Enter) and back; page content comes back fenced
as untrusted data; only http(s) addresses; no LLM call. Offered only when the app started the bot
browser (JARVIS_BROWSER_CDP) and the browser feature is on; meanwhile fetchWebPage, which has no
address guard, is not offered. Eval evals/test_bot_browser_tool.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:18:05 +09:00
javis-bot
46e4e75d05 feat(voice_worker): a short voice prompt and one judgement before each v2 answer
- compact_prompt: the v2 persona in a tenth of v1's words, answers as short as
  possible, the full tool guidance only on turns with a data tool. A greeting's
  reply prompt went from about 4,500 to 1,400 tokens on Strata.
- intent_judge: the tool router judges whether the turn needs tools; the
  planner runs only when it does. prepare_intent runs the pick ahead on a
  partial transcript (worker message prepare), and the reply reuses it.
- Without stream_browser the screen-share tools are not offered.
- The time context carries the 12-hour form (10:37 was read as pm).
- A <tool_call> block written as the reply is parsed as a call, not spoken.

Evals (Qwen3.6 on Strata, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1):
221 passed, 4 failed (known failures); new evals/test_voice_reply_length.py
(v2 answers 234 chars against v1's 416) and test_reply_language_default.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:25:15 +09:00
javis-bot
7623d6d002 feat(voice_worker): the v2 brain answers in Korean unless asked for another language
The brain passes default_language (worker --reply-language, default Korean) to
the reply engine; unlike the output_language lock it gives way when the user
explicitly asks for a language. Tools get the ko code (--tool-language).

Eval evals/test_reply_language_default.py (Qwen3.6 on Strata): 6/6, English
questions answered in Korean, an explicit English request in English, and the
control without the default answers English in English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:12 +09:00
javis-bot
f892a1dce9 feat: answer calls in the voice channel through the response policy
- conversation.rs: the loop between the core's events and the rest.
  Finals go through call detection and the response policy; answers are
  asked of the brain and spoken from the app's queue, an interruption
  cancels only that answer, a voice stop silences everything, and missed
  requests are kept without text (last 100). Tested with a fake voice side.
- Core: answer(user, text) adopts a playback request the worker speaks
  under the same id, cancel(id) stops one request, ask_addressed and
  ask_rank pass the conversation's questions; new SpeakingStopped,
  Addressed and Ranked events.
- Worker client: reply, addressed and rank messages and their results; a
  reply with no answer, refused or not sent ends its speech as failed.
- Voice worker: addressed and rank messages on their own threads, beside
  a running reply; a spoken reply cancelled while the brain is answering
  is not spoken. judges.py holds the two short LLM judgements with the
  heard text fenced as data; eval evals/test_voice_judges.py.
- Specs (conversation, discord core, voice worker), docs/llm_contexts.md
  and the roadmap updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:26:47 +09:00
javis-bot
3ec430fd25 fix(reply): known user location reaches getWeather; retries are not blocked by the previous reply
- Drop the engine's deterministic Korean weather shortcut. It bypassed the
  chat model and the tool runner, so a location the user had stated could
  never be used. The tool already returns one clean Korean sentence.
- getWeather and the tool-argument prompts pass the place the user has
  said they are (conversation or memory) and leave the argument empty
  otherwise, so the detected location is not copied into it.
- getWeather retries a "district, city, country" name with its leading
  parts dropped when the geocoder finds nothing.
- The planner gets the live time/location line and plans
  searchMemory topic='where the user lives' only when the location is
  unknown or disabled.
- The per-reply repeat guard counts only this reply's tool messages, so
  calls carried over from the previous reply no longer block a retry the
  user asks for.
- Memory extractor: the context block sits after the examples and the
  no-location example uses a different query, so a location already in
  context is not asked for again. New counterpart eval checks that it is
  still asked for when the context has none.
- Tool retry eval fails every call in turn 1, so a model that retries by
  itself inside turn 1 still exercises the user's "try again".
- Specs (planner, reply), docs/llm_contexts.md and the roadmap updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:55:45 +09:00
javis-bot
69a884b182 test(evals): run the evals against any server named by EVAL_JUDGE_BASE_URL, including /v1
Ten eval files pinned their reply engine to http://localhost:11434 and the
judge only spoke Ollama's /api/tags and /api/chat, so pointing the suite at
Strata (OpenAI-compatible /v1 only) skipped or broke most evals.

- MockConfig and every eval config read EVAL_JUDGE_BASE_URL.
- The judge availability check reads /v1/models and the judge call goes
  through jarvis.llm when the URL ends in /v1; the Ollama path is unchanged.
- run_evals.sh probes /models for a /v1 URL.

The intent-judge and listener evals stay pinned to gemma4:e2b on local Ollama.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:49:08 +09:00
javis-bot
4621e6c1fd fix(chat): the delete consent check knows the pending message and reads the whole reply
The consent classifier is now told which message and channel are pending, so a yes for another
message ("yes, delete message 22" while 21 is pending) is a no and cancels the request. A reply
longer than 600 characters is no longer cut and judged by its start (which could drop a later
"don't"); it counts as no without a model call.

Eval evals/test_chat_delete_consent.py gains the different-message and long-reply cases:
17/17 on qwen2.5:7b, three runs. The previous prompt said YES 3/3 to
"응, 일정 채널에 있는 거 지워" while a message in 잡담 was pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:26:58 +09:00
javis-bot
0182823592 fix(chat): delete needs the person's own yes; chat action ids never repeat
- chatChannels delete with confirm=true now runs only in the person's very
  next turn and only when their own message of that turn agrees, checked by a
  small fail-closed YES/NO call on the router model chain (the message fenced
  as data). A refusal, question, other topic or error cancels the request;
  an unconfirmed request lapses after one turn. The reply model's confirm
  flag alone can no longer delete.
- chat_action ids carry a per-run nonce and the session, so a late answer
  meant for an earlier connection (or worker run) never completes a newer
  action.
- Tests: agree, refuse, other topic, lapse, classifier failure, late answer
  across reconnects. Eval test_chat_delete_consent: 12 of 12 on qwen2.5:7b,
  3 runs. Spec and llm_contexts updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:08:35 +09:00
javis-bot
fe4e296361 feat(chat): chatChannels tool for the registered Discord chat channels
- Tool (chat_channels.spec.md): list, read, search, send, edit, react,
  delete, pin, unpin. Offered only when the v2 brain gives the turn a chat
  backend, and then on every turn. Channel by ID, name, part of a name or a
  name inside the text; a message ID finds its channel; planner forms of the
  action ("send_message") work. Read and search results are fenced as
  untrusted data. delete only reports until called again with confirm=true
  in a later turn of the same person.
- Worker: chat_channels from the app, ChatBridge reads/searches the history
  store and sends chat_action, waiting for chat_action_result (15 s timeout,
  failed on session close); per-person reply turns and pending deletes.
- App: channel switches edit/react (on) and delete/pin (off); chat_channels
  sent on worker ready and after login; chat_action runs send, edit (own
  messages only), react, delete, pin, unpin and answers with ids or the error.
- Engine: chatChannels left out of the catalogue without a backend; the
  "router returned everything" check now tolerates that.
- live_discord: edit own, refuse editing the speaker's, react, pin (Discord
  refuses in a voice channel's chat, reported as an error), delete, refuse an
  unregistered channel; passed in 109.4 s.
- Eval (qwen2.5:7b, 3 runs): 10 of 12; never obeyed the "delete everything"
  message and never deleted in the same turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:01:33 +09:00
javis-bot
b3dcd12ec9 fix(voice_worker): keep history and the brain's output scoped to who was there
- A final is stored under the channel, speaker name and time fixed when its
  end was decided. Decoding takes a while; a move, rename or late joiner
  meanwhile used to change where and when it was said, and who heard it.
- At start the worker closes presence a killed run left open
  (HistoryStore.close_open_presence), so a person who is gone no longer
  hears new speech after a restart.
- run_reply_engine(legacy_memory=False) skips v1's single-user memory (warm
  profile, extractor, diary, graph, digest); the v2 brain uses it so another
  user's profile or diary never reaches the prompt.
- While a brain reply runs, stdout and stderr are muted (also helper threads
  such as the web fetch pool); the worker logs through voice_worker/log.py to
  the original stderr.
- Live check: Ollama's /v1 fixes the context at 2,048 tokens and cut the
  prompt in earlier runs; with an 8k tag the chat-record question was answered
  4 of 4 and nothing reached the output. Noted in llm_contexts and the eval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:28:25 +09:00
javis-bot
864b2e7fdb feat(voice_worker): record history from the app and answer reply requests with the brain
Worker (voice_worker.spec.md "History and brain"):
- --history <file> keeps the conversation history store; --brain hosts the
  jarvis reply engine (JarvisBrain, settings point ollama_base_url at /v1).
- voice_join / voice_leave / member_join / member_leave keep presence; the
  session end closes it. Non-empty finals are stored under the speaker's name
  in the current voice channel.
- chat_message / chat_edit / chat_delete keep registered chat channels.
- reply builds the asker's scoped history context, runs the brain on a thread
  (one at a time), answers reply_text, stores a voice answer, and with
  speak=true says it under the same id. memory settings in hello and
  memory_config. brain_unavailable without --brain.

App (discord_core.spec.md "History reporting"):
- WorkerClient sends the new messages; the core reports the voice channel
  and its heard members after a join, a move and every worker reconnect,
  member joins and leaves, voice_leave when a session ends, and every message
  of a readable registered channel (its own too, with `to` on replies), edits
  and deletes.
- live_discord runs the worker with --history and checks the store at the
  end: 8 finals under the speaker's name, the edited chat message only,
  Jarvis's own messages, 5 presence intervals, none left open (passed, 107.8 s).

Eval: history recall gains a chat-record case and fails a reply that reads a
record stamp out (field run with qwen2.5:7b did once in four).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:16:24 +09:00
javis-bot
54a465948a feat(memory): v2 conversation history store and its context in the reply prompt
- src/jarvis/memory/history.py keeps voice transcripts, Jarvis's answers,
  registered chat channel messages (with edits and deletes) and voice channel
  presence in one SQLite file. Every read is scoped to the asker: own records
  always, chat channels and others' words (only while the asker was in the
  same voice channel) behind settings, plus an optional since and off.
- Search is language-agnostic: Unicode words, prefixes keeping 60% of the
  word, rarer words weigh more, newer wins ties. 100k records search in
  24-61 ms here. context() returns the recent window and the matches cut to a
  token budget, matches at most half so chatter cannot push out the record
  the question is about.
- run_reply_engine(history=...) puts that context into the system prompt
  under "Information shared in previous conversations", fenced as data.
  evals/test_history_context_recall.py: with the block qwen2.5:7b answers the
  stored day in Korean and English (3 of 3 runs), without it the day is unknown.
- Spec history.spec.md, registry entry in CLAUDE.md, reply.spec.md,
  docs/llm_contexts.md, docs/v2/memory.md and roadmap updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:45:33 +09:00
javis-bot
903e0eb9de fix(reply): stop models answering past an available tool
M0 on Strata caught both v2 models skipping a tool they had been given:
the recommended model made up "24 degrees, sunny" instead of calling the
weather tool, and the fast model said "timer set" without calling the
timer tool (3 of 3 each). qwen2.5:7b does the same through the engine.

- Prompt: TOOL_HONESTY_RULES in the persona (live facts only from a tool
  result this turn, actions only claimed after a tool reports success, no
  answer text next to a tool call), restated at the end of the system
  message by tool_honesty_reminder() with the turn's real tool names.
- Engine: skipped-tool guard. When a native-tool model answers before any
  tool has run and the plan names a callable tool as a concrete call, the
  answer is dropped and the plan steps are direct-executed, then the model
  answers from the results. Only literal, schema-valid steps run (regex
  fast path, no LLM repair); otherwise the held-back answer is delivered.
- Planner: the AVAILABLE TOOLS lines now carry argument names
  (planner_tool_catalog), so concrete steps use the tool's real keys.
- Eval evals/test_tool_skip_confabulation.py; on qwen2.5:7b, 3 runs each,
  available-tool cases went from 0/12 to 9/12 calling the tool.
- M0 kit: tools_fast / tools_recommended stages (run_m0_tools.bat) rerun
  tool calling before and after the fix on the real models, loading the
  production system_prompt.py and planner.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:05:40 +09:00
javis-bot
c4abf63f38 Add Discord-native hybrid front-end for Jarvis (bot + bridge)
Some checks failed
Release / semantic-release (push) Successful in 59s
tests / Unit tests (Linux, Python 3.11) (push) Successful in 13m45s
Release / build-linux (push) Failing after 7m47s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Transform isair/jarvis into a Discord-controlled voice assistant running on
the Ubuntu VNC desktop, keeping the mature ~39k-line Python brain intact.

- bot/ (Node + bun, discord.js): /자비스 slash commands (ephemeral),
  voice channel join + voice receive/playback, pluggable VNC screen broadcast
  (selfbot live / noVNC / screenshot)
- bridge/ (Python, Flask): wraps jarvis STT + run_reply_engine + Piper TTS
  behind a thin localhost HTTP API
- .env.example, scripts/ (start_bridge/start_bot/dev), README rewrite,
  docs/language-comparison.md and docs/vnc-xfce-setup.md

Language decision: hybrid (Python brain + Node/bun Discord layer) because
Discord blocks bot video; native screen broadcast only works via a Node
selfbot library.
2026-06-09 14:51:05 +09:00