Commit Graph

138 Commits

Author SHA1 Message Date
javis-bot
218e7734be fix(v2): botBrowser refuses a negative volume level
level=-10 was read as 10 (the sign was dropped) and unmuted the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 01:02:01 +09:00
javis-bot
decd7ae477 fix(v2): YouTube no longer stops 45 s in, consent and the bot's video keep their tools
From the live conversation on 0.1.40 (2026-11-18):

- The bot browser told pages a program drives it (navigator.webdriver), and
  YouTube stopped every video about 45 s in with "문제가 발생했습니다"
  (ump.spsrejectfailure, reproduced twice on the game PC). Chrome now starts
  with --disable-blink-features=AutomationControlled, as the M0 video check
  already did; the live Chrome test checks webdriver is false.
- "네." to "계산기를 실행해도 될까요?" got webSearch and botBrowser from the
  router, so the consent call was refused and the calculator never started.
  Waiting tools: pcControl stays on offer while a request asked in the
  previous reply waits, botBrowser while it holds the asker's page.
- "소리 반으로 줄여 줘" for the bot's video worked on the person's own Edge
  window. botBrowser gets a volume action (media elements and YouTube's player),
  the page shows its volume, its description names its actions first (the part
  the planner sees), pcControl's says the bot's browser is not among its
  windows, and a turn whose plan has no step while a tool holds something for
  the asker gets the one-step nudge for it.

Evals (qwen3.6 on the game PC): volume 0/3 before, 8/8 after; "네." consent
10/10; pcControl 10/10; botBrowser 16/18 (the NewJeans play case fails 0/6
before this change too). Unit: Python 2542 passed, 13 failed (the same 13 as
before); app 348 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:47:38 +09:00
javis-bot
7539be07a5 feat(pc): pcControl brain tool and the worker's pc_action bridge
The brain operates programs through the app (pc_control.spec.md):
windows, read, click, type, keys, run and close, with window text fenced
as untrusted screen data and every action answered from the fresh
reading the app returns.

- Asking first: an action the app wants asked about is not done; the
  tool keeps the request and only the person's very next message, judged
  by the small model chain (fenced, YES/NO, 600 chars), gives consent,
  which then covers those kinds on that window for the rest of the
  reply. The model's confirm flag is not what decides.
- Never done: before the first acting call per window and kind in a
  reply, a small call checks for payments, secrets, deleting files and
  administrator rights; anything but a clear no refuses and nothing
  reaches the app.
- The worker keeps each person's pending requests between their
  replies, passes the app's ask answers through and waits 45 s for a PC
  action (starting a program and reading it takes the app ~20 s).
- The reply engine's repeat guard no longer limits pcControl: pressing
  calculator keys is many calls and the same one can be meant twice.
  Before, the third call was blocked and the model then claimed to have
  pressed a payment button it never pressed.

Eval evals/test_pc_control_tool.py on Qwen3.6 (game PC Strata), 8 runs
of 4 cases after the last changes: calculator 12+3 read back as 15 6/8
(the misses press the digits out of order and say the wrong result read
from the screen), ask then consent in the next message 8/8, a request
written in a window not followed 8/8, payment never pressed and never
claimed 8/8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:42:35 +09:00
javis-bot
b58fc149d4 perf(voice): ask the done-by-user question while the user still speaks
The done-by-user guard asked its first yes/no question after every
follow-up answer was made, adding a short LLM call (~0.4 s) before
Jarvis spoke. The question depends only on the user's words, so the
voice worker's prepare now asks it on the partial transcript after the
tool pick (prepare_done_by_user, follow-up turns only) and the guard
uses that judgement (prepared_done), asking only whether the answer
refuses when the user said they did it. Without a prepared judgement,
or when it failed, the guard asks for itself as before. The reply does
not wait for the judgement, only the guard does.

Measured on Qwen3.6 through the game PC's Strata, 12 pairs of short
follow-up questions in alternating order: brain time median 1.06 s
before, 0.69 s after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:07:04 +09:00
javis-bot
3e4974f3bd fix(reply): done-by-user guard keeps tool calls, other people's lines and started tools safe
- A native tool call message with text is not an earlier answer: leaving it out
  orphaned its role=tool result. Only plain assistant answers are left out.
- The guard is off once any tool run started in the reply, whatever it returned:
  an MCP call with an empty result was not in invoked_tools_history, so a second
  try could run it again.
- HistoryContext.answers marks the lines that are Jarvis's answers to the asker
  (to_id is the asker); only those leave the history, not someone else saying the
  same words.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:55:29 +09:00
javis-bot
9c1e82a890 fix(reply): answer again without earlier answers when the user did it and the answer refuses
Field turn 2026-11-18: Jarvis said speed control is not supported, the user changed
the speed in the app and said "I adjusted it, so say something", and Jarvis refused
again, often in other words. Prompt wording did not move it (five rules, two history
shapes tried).

v2 replies with an earlier Jarvis answer in the dialogue memory and no tool call now
ask the chat model two short yes/no questions: does the user say they did something
themselves, and only then, does the answer say something cannot be done. On two
yeses the answer is made once more with Jarvis's earlier answers (dialogue memory and
the history lines with the same text) left out of that reply's context.

Real model (qwen3.6 35B on Strata), field case with history and dialogue memory, 20
same-prompt pairs: refused 12/20 before, 1/20 after; "what did you say? say it
again" still said it again 20/20 before and after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:42:15 +09:00
javis-bot
ac633c63ca fix(voice): no treble boost before transcription
The 12 dB shelf above 1 kHz from 0.1.34 made the user's speech worse.
Re-transcribing the 34 live recordings of 2026-11-18 (17:32 to 19:53)
with the shelf reproduces every live text; without it "왜 내 방해를"
comes back instead of "올해 내 방해를" (the user said "난 왜 라고
했어" right after), "알아서 좀 조정해 놨는데" instead of "아라해석도
좀 조정해놨는데", and "넌 뭐하는 애야?" instead of an empty text. No
utterance got clearly better with it. Reverts 9f62bfb and 04c23f9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:04:24 +09:00
javis-bot
9f5f3d891f fix(voice): spell Korean numbers out before MeloTTS reads them
MeloTTS's Korean front end read "60%" digit by digit ("육영퍼센트") when
the Windows MeCab build did not tag what followed as a bound noun, and its
sentence splitter broke "4.2미터" at the dot ("사. 이미터"). The KR child
now spells decimals with 점, drops thousands commas and gives every other
whole number its Sino-Korean reading, leaving numbers before native
counters (시, 개, 마리, ...) to MeloTTS's own counter rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:32:46 +09:00
javis-bot
9f62bfb059 fix(voice): raise the consonant band by 12 dB before transcription
A dark microphone or a speaker who swallows first consonants and
sentence ends loses the band above 1 kHz first: in the live
recordings "자비스" came out as "아비스"/"다비스" and "서울" as "어우".
loudness_equalise now raises it with a one-octave shelf at 1 kHz.

Measured with large-v3-turbo: 25 sure live utterances 5.0% -> 3.4% CER,
FLEURS Korean test split (382 clips, held out) 5.16% -> 4.98%. The real
noise check stays at 108 clips with no text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:09:05 +09:00
javis-bot
88ee5ec2cb feat(weather): the app's log names the service that gave the weather
The source line (KMA, open-meteo or met.no for now; KMA and open-meteo
day counts for a forecast) only went to debug_log, which is off by
default and muted while the worker's brain answers, so a live answer
such as "흐림" could not be traced to its service. It now goes to the
llm log tab without the place or the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:10:06 +09:00
javis-bot
c7655c58f4 feat(weather): detail on request, 소나기 in 단기예보, weekday counts in the schema
- detail (only when the user asks for more detail) adds the humidity, the
  wind in m/s and the rain amount: now the last hour's rain (KMA RN1,
  open-meteo precipitation), a 단기예보 day its mean humidity and strongest
  wind, an open-meteo day its strongest wind and rain total.
- 단기예보's PTY 4 (소나기) was missing, so a showery afternoon under a
  clear sky came out as 맑음.
- days_ahead's description lists each coming weekday's count from today:
  with the new argument qwen3:8b counted a Wednesday's weekend as 2.

Evals: evals/test_weather_detail.py, test_weather_forecast_days.py,
test_weather_district_place.py (qwen3:8b, 27/27 over three runs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:21:46 +09:00
javis-bot
9ba26c5bac feat(weather): districts such as 부산 연제구 found on OpenStreetMap
open-meteo's geocoder knows cities only, in Latin spellings, and matches a
non-Latin name loosely: "부산" gave a village called Pusan 140 km from
Busan. A non-Latin name now goes to OpenStreetMap's Nominatim first (any
language, down to districts), and a Latin name open-meteo does not know
goes there before the LLM romanisation and the wider-area retries. The
answer names the place in the user's own words, and KMA gets that spot's
grid point. Eval: evals/test_weather_district_place.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:08:15 +09:00
javis-bot
17927b6450 feat(weather): KMA forecasts for a later day (days_ahead, days)
getWeather takes days_ahead (days from today, up to 10) and days (a span,
2 for a weekend). Inside Korea with the app's KMA key: 단기예보 for today
to 3 days ahead (morning and afternoon weather, lowest and highest, highest
chance of rain), then 중기기온 and 중기육상 for the nearest of 32 KMA cities
and its region. Days KMA does not give, and every KMA failure, come from
open-meteo's daily forecast. days_ahead 0 with more days gives the weather
now, then each day. Eval: evals/test_weather_forecast_days.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:57:12 +09:00
javis-bot
282f114e60 feat(weather): KMA current weather inside Korea with the app's stored key
The 기능 screen's 기상청 날씨 card stores the KMA API hub key encrypted
(kmaApiKey, masked in logs). On start the app passes it to the worker in
JARVIS_KMA_API_KEY; the worker takes it out of its environment before any
child process starts. Inside Korea getWeather reads 초단기실황 and the
coming hour's sky from 초단기예보, and falls back to open-meteo, then
met.no, on any KMA failure. The key is in KMA's URLs, so failures are
logged by their kind only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:31 +09:00
javis-bot
af1276cf6b refactor(search): remove the Gemini search backend
The user asked to drop it: nobody had the Gemini CLI login or a key set up,
and Exa then Wikipedia cover the voice use. webSearch is now live Chrome,
Exa, Brave (opt-in), Wikipedia. Removes the CLI/REST backends, the
GEMINI_* settings, the Docker login mount and checks, and updates the specs
and docs/llm_contexts.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:24:11 +09:00
javis-bot
59a9277613 fix(weather): met.no sleet is sleet, not freezing rain
met.no's sleet symbols were mapped to WMO 66/67 (freezing rain), so the
answer said 어는 비. They now map to the WMO 4677 rain-and-snow codes
68/69 and 83/84 (진눈깨비, 진눈깨비 소나기).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:22:42 +09:00
javis-bot
2238ddbe42 feat(search): search with Exa, Gemini only with a login, title-guarded Wikipedia
DuckDuckGo now serves only its bot-check page (game PC and host alike), so
every search fell to Wikipedia full-text search, which picked unrelated
pages (박진영 for songs for a rainy day, 대한민국 for a graphics card
ranking). Live 2026-11-18 answers took 9-15 s and were wrong.

- Exa's free MCP endpoint (no key) is the first provider after the
  live-broadcast Chrome; it returns page text, so no fetch is needed.
- Gemini runs only when Exa found nothing and a CLI login (oauth) or key
  (apikey) is present; without a login the CLI is no longer started
  (about 2 s per search).
- Wikipedia is used only when the page title is named in the query;
  otherwise the honest nothing-found envelope.
- Letters of a script the user did not use are dropped from the query
  (the model searched '사과와 배 중哪个가 더 단가요').
- DuckDuckGo removed; spec, docs and tests follow the new order.
- weather: any forecast failure (not only requests errors) tries met.no.

Real model (qwen3.6-35b-a3b, game PC), real network: graphics card
ranking, apples vs pears, rainy-day genres and black holes all answered
correctly in 3.7-5.2 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:11:28 +09:00
javis-bot
680d4fbdc1 fix(weather): fall back to met.no when the open-meteo forecast does not answer
Live 2026-11-18 the open-meteo forecast timed out three times (geocoding
worked) and every weather question ended in 'service unavailable'. The
forecast now waits 6s instead of 10s and then asks met.no (no key),
mapping its symbol to the WMO code the Korean sentence uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 01:59:49 +09:00
javis-bot
44387685af fix(app): log the card's real free memory, and short answers get their line too
Free memory is nvidia-smi's memory.free in the app and the voice worker, so the
driver's reserved share no longer counts as free. A card reading still running
when an answer ends finishes on its own and then logs, so an answer shorter
than one reading is logged too, and the answer still never waits for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
f1600f09c2 feat(app): log the graphics card's free memory after the LLM starts and while each answer is written
The app logs how much of the card is free once the model answers, and the
voice worker reads the card once a second while an answer or a scheduled task
is written and logs the least free memory seen (no text, llm tab), so a slow
answer can be matched to a full card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:20 +09:00
javis-bot
8c61d7a821 perf(llm): short one-shot calls say so to Strata
Every call that no later request continues (call_llm_direct,
call_llm_streaming, the warm-up and the v1 intent judge) now sends
"strata_checkpoint": false on the OpenAI-compatible path. Strata's
Qwen3.6 engine then gives that call's session away first, so with two or
more requests at once a reply's tool turn keeps its session through the
short calls made between its steps. The reply loop's chat_with_messages
is not marked (its next step continues it); the Ollama API gets no new
field. With one request at once nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 07:06:25 +09:00
javis-bot
4d0ae91e8a perf(reply): v2 tool turns get the tool guidance in short, and a shorter reply-language default
Strata reads every reply prompt from the start in 2,048-token steps. The v2
compact prompt carried v1's full tool guidance and constraints (about 900
Qwen3.6 tokens) on every turn with a data tool, and the reply-language
default (about 120 tokens) twice.

COMPACT_TOOL_GUIDANCE keeps the same rules (act instead of refusing, facts
only from the result, look up named things, auto-derived and self-contained
arguments) in about a quarter of the tokens, without English examples; the
reply-language default says the same in half. v1's prompts are unchanged.

Measured on Strata (Qwen3.6, 20 lines of recent conversation): a weather
turn's prompt 3,436 -> 2,722 tokens, a search turn's second call
4,096 -> 3,383. Evals (compact prompt + intent judge, Qwen3.6): 24/24
before, 30/30 twice after (weather location, follow-up arguments, tool-skip
confabulation, search language, reply language, Possessor fidelity, reply
length, multi-turn context).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 21:13:08 +09:00
javis-bot
7da5bca783 fix(reply): a one-step turn whose model skips the tool twice is planned and guarded
After the one nudge, a second answer without the tool no longer goes out
as it is: the turn takes the planned path (the planner runs then, one
extra call only on this failure) and the skipped-tool guard runs the
planned tool, so a made-up "24 degrees" answer cannot reach the user. A
plan that needs no tool (a question back to the user) still lets the
answer go out.

Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs):
the five tool-skip/weather/screen/follow-up/router files 72/84 (71/84 on
0.1.20, same known failures); Possessor director-then-filmography 0/3,
also 0/3 on 0.1.20 (qwen3:8b stops after one search; webSearch turns are
still planned).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:09:11 +09:00
javis-bot
a6932d9b16 perf(reply): v2 turns with one one-step tool skip the planner
When the router's only pick is a tool that does its job in one call
(getWeather, logMeal, fetchMeals, screenView) and the chat model calls
tools natively, the planner no longer runs: on Strata it averaged 2.5 s
per tool turn (2026-11-16 16:45-16:58 engine log, 9 calls) before the
answer could start. Search, browsers, schedule, notes, chat channels and
picks of two or more tools still plan. The rule reads the router's pick,
not the words, so it is language-agnostic.

Without a plan the skipped-tool guard has nothing to run, so these turns
get one nudge instead: an answer given before any tool ran is dropped and
the model is asked to call the tool (or ask for a missing detail).

Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs of
tool_skip_confabulation, weather_autoderive_location, screen_view_tool,
followup_supplies_missing_tool_arg, tool_router_implicit): 71/84 before,
72/84 after, same known failures; planner calls 14 -> 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:00:36 +09:00
javis-bot
cda8db7fce fix: play on YouTube in one step, Korea as the default location, searches in the user's language
- botBrowser `play`: searches YouTube for the words as said, opens the first video (not an ad,
  without the playlist) and starts it, in one call (an address given opens that page); `type`
  also takes `query`. Live 0.1.19: the model stopped on YouTube's home page or the results, a
  retry was refused as a third call, and the answer still said the video played.
- `brain.location` (기본 위치, default 대한민국) goes to the brain as `location_name`: the time line
  says it without GeoIP (no database on the game PC), and getWeather without a place uses it.
- Planner and webSearch keep search words in the user's language (never translated into
  English), so recommendations fit where the user is.

Evals (qwen3:8b, 3 runs): play ending on a playing video found with the words as said 0/6 -> 5/6;
Korean searches for Korean requests 6/9 -> 8/9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:26:30 +09:00
javis-bot
420c9c73f3 fix: stopped sentences recorded only as far as heard, search words as plain query, paused video said so
- Playback sends how long the clip playing when a request was stopped had played (tts_played
  last_ms); the worker cuts the last sentence of that recording to it.
- botBrowser open takes the search words as plain `query`: the tool puts them, encoded, into the
  address's empty search parameter or types them into the site's search box. The description
  tells the model to search this way and never put words into an address itself (live: 아이유
  hand-encoded into 아이우). Eval (qwen3:8b, 3 runs): right words reach YouTube 1/3 -> 3/3.
- When a page's main video does not play (also after the tool pressed play, or it failed to load),
  a note after the page says it is not playing and not to say it plays. Eval: a paused video
  reported as playing 1/3 -> 0/3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:12:04 +09:00
javis-bot
ec2672cd4a fix: recordings keep only what was played, newest waiting request kept, video pages opened play
- Playback tells the worker how much of each request was played (tts_played); the worker keeps a
  spoken answer's recording only that far (an answer stopped mid-way no longer keeps its unheard
  rest, and is marked stopped); nothing played or never reported: no file.
- Old recording day folders are also removed with the first recording of each new day.
- response.interrupt off, latest: a request whose final comes after a newer one's is replaced
  instead of replacing it.
- botBrowser presses play on a page's paused main player also after open, not only after a click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:56:40 +09:00
javis-bot
72518ca50e feat(app): addressed calls follow who is in the voice channel
Alone with Jarvis, whatever the speaker says is answered without asking the judge. With others
present the judge is told how many people are in the channel and how many may call Jarvis: with
one other person most of what the speaker says counts as said to Jarvis (unclear is yes); with
more, or not known, the question is the group one it was before. Live 2026-11-16: alone in the
channel, questions without the wake word got no answer because the judge assumed a group.
Eval (qwen3:8b, pair cases): 3/7 before, 6/7 after; group cases unchanged.

Also: the protocol test fixture's brain takes the trace argument (the test failed since traces).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:43:10 +09:00
javis-bot
cbe898d3d0 feat(app): log screen saves (shown lines, conversation, voice recordings), stop-or-finish on a new request, video pages that play
- 로그 screen: "보이는 글 txt로 저장" (logs_save_text), a 대화 view that saves the days'
  conversation with every trace in full (history_export), and a 목소리 view with the
  voice.record switch and the days' recordings as a zip with an index (voice_export).
- Voice worker: with JARVIS_RECORDINGS_DIR (voice.record, off by default) each final and each
  spoken answer is kept as a WAV with an index line; day folders older than 7 days removed.
- response.interrupt (default on): off, latest/important finish the running answer first.
- Bot browser: Chrome lets pages start video and sound; botBrowser reports each page's media
  state, waits for a player to start after open/click, and presses play on a video page a click
  led to when its player stayed paused (live: a clicked YouTube video stayed paused).
- webSearch reads `query` when `search_query` is missing (live: both searches of an answer failed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:27:57 +09:00
javis-bot
6fc3ad3d57 feat(worker): answer traces stored from the question on and filled in as the answer is made
A trace is now kept the moment the request arrives (at the time it was
asked, ms -1), its steps are stored as the engine adds each one, and it
ends with the answer. Traces a killed run left unfinished are closed at
start. The 대화 screen reads a trace still being made as pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:03:45 +09:00
javis-bot
a20ebe4eae feat(brain): keep how each answer was made (records, tools, plan, thinking, tool calls)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 03:00:10 +09:00
javis-bot
27eb1315b9 fix(tts): MeloTTS speaks up to 1.25 itself so fast speech sounds natural
Stretching all of the speed-up from 1.0 kept every sound but sounded
awkward (UTMOS 3.13 at 1.5, against 3.73 for MeloTTS's own 1.5), while
MeloTTS's own speed past 1.25 drops sounds (sentence-final "요" and first
syllables, 4.6% character errors at 1.5). MeloTTS now speaks up to 1.25
and the WSOLA stretch does only the rest, with a 5 ms search: 2.6% and
UTMOS 3.56 at 1.5 through the real child. Version 0.1.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 00:06:52 +09:00
javis-bot
dbe5c617bc feat(tts): change the speech speed without a restart and keep fast speech clear
MeloTTS's own speed shortens every sound and slurred fast speech (Whisper
large-v3 on five Korean sentences, three renders each: 5.5% character
errors at 1.5 and 20% at 2.0, against 1.4% at 1.0). The MeloTTS child now
speaks at most at its natural pace and makes faster speech with a WSOLA
time stretch that keeps the pitch (1.9% at 1.5, 2.2% at 2.0, about 15 ms
per 4 s of speech).

A voice.tts.speed saved while the services run goes to the worker at once
(new tts_speed message) and applies from the next sentence; the client
sends it again whenever the worker connects again. Version 0.1.4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 23:39:04 +09:00
javis-bot
53bccb620d feat: screen viewing, Jarvis sees the whole screen, a monitor or a window for registered people
- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
  title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
  viewing"), decides for the asker (only people registered for PC control), never picks, lists
  or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
  minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
  OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
  not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
  game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 17:02:59 +09:00
javis-bot
bd7b69d4f4 fix(browser): the bot opens its pages in windows of their own behind, so it never takes the focus
On the shared screen a tab brought to the front makes Chrome bring its window forward and take the
keyboard focus from the game or program the user is in (seen on the game PC). The botBrowser tool
now opens each person's page in a window of its own, behind and at the place and size of the bot
browser's window (Target.createTarget with newWindow and background), and closes it again if the
connection to it fails. While the browser runs on the virtual monitor, the app looks for new Chrome
windows every 200 ms and settles each like the first (tool window, fitted to the work area, without
activating it), the newest in front of the older ones, so the shared screen shows the page the bot
opened last.

Checked on the game PC: the bot's window loading a public page comes in front of the first, both
out of the taskbar and Alt+Tab, the focus staying in Discord also with the user idle for hours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 16:15:36 +09:00
javis-bot
324fd2ea0a feat: news notifications for the background collection and a collectedNews tool
Each collection target can announce new items at once or in a daily digest
(off by default) in collect.notifyChannel, as titles and links only, with no
LLM; each post is recorded in the activity log. The brain gets a read-only
collectedNews tool (every target's newest items by default, fenced as
untrusted web content), offered with the app's collection file and the
collection feature, and available to scheduled tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:22:54 +09:00
javis-bot
a5bd25b86f feat: personal notes, each person's standing instructions for how Jarvis answers them
A personalNotes tool (add, list, delete the asker's own notes) offered only
with a notes backend. The app keeps the notes in config/notes.json and sends
the asker's with every voice, chat and task reply; the reply engine puts them
in a <person_notes> fence as that person's preferences, which never permit an
action or a tool. The worker's schedule bridge becomes a shared app bridge
for schedule_action and note_action. A 사람별 지침 screen edits and deletes
any note; features.notes turns the tool and the notes off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:10:53 +09:00
javis-bot
04b603facd feat(app): an activity log of what Jarvis did (chat actions, schedule deliveries)
Each chat channel action the brain's tool asked for and each schedule
delivery is recorded in activity/<day>.jsonl by kind, target, who asked and
outcome, never its text, kept 30 days. The worker's chat_action carries the
asker. A 한 일 screen lists the newest entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 16:51:18 +09:00
javis-bot
ea2d3ba958 feat: reminders and scheduled tasks (schedule tool, app store and delivery)
The brain gets a schedule tool (add, list, delete the asker's own reminders,
timers and tasks) offered only with a schedule backend. The voice worker asks
the app through schedule_action and runs fired tasks as reply via task with
read-only tools, no MCP and a fresh short-term memory. The app keeps the
schedules in config/schedules.json, fires them every 5 s, speaks them when the
person is in Jarvis's voice channel and otherwise posts in the chat channel
they asked in (or schedule.chatChannel for voice). A feature switch and a
예약 screen list and delete everyone's schedules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 16:27:42 +09:00
javis-bot
3d58cf8ab1 feat(voice): in a game, partial transcription waits while an answer is written
- the app's ready message says whether the game stage runs (llm ready game=true); the worker then
  holds partial passes while a reply is being written and runs them again, the newest audio per
  speaker, when no answer is; finals never wait; without a game nothing changes
- M0 gamemode_tune decision: the game placement answered 1.7 times faster with partials held
  (64.9 against 37.5 tok/s with a partial pass every 2 s)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:58:11 +09:00
javis-bot
aa6d950af6 fix(app): performance stage switches survive failures, reconnects and late signals
- the normal stage reads other programs' memory afresh (the card after the LLM stopped, less the
  STT worker's model) and keeps it; the game stage counts from the latest normal reading, so the
  game is not counted against its own reserve
- a restart that fails brings back the stage that ran before and tells the brain ready; the
  failed stage is tried again after 60 s, three times at most; when nothing comes back the brain
  is told paused so requests fail at once
- the worker client keeps the latest llm state and sends it first on every connection
- switches are numbered: llm switching carries an id, llm_idle names it, and a late idle of an
  earlier switch no longer lets a new one through
- a request that waited 180 s for a switch fails without calling the brain
- the dashboard shows the stage that runs also on a fallback placement without experts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:34:24 +09:00
javis-bot
7e096b449e feat(app): performance stages move the LLM between the normal and the game placement
- while the services run, game mode decides the stage (perf.rs): a game means the game stage at
  once (or pause with game.pauseInGame); its end brings the normal stage back after
  game.revertSeconds (default 30)
- a change tells the voice worker the LLM switches (llm switching, with a notice), waits until
  the brain has written the answers in progress (llm_idle, at most 2 minutes), restarts the LLM in
  the stage and tells the worker it is ready or paused
- the game stage runs at 16K context, one request at a time, half the engine's CPU threads and
  the game placement (reserve game.reserveGb, default 8, else 2 GiB, else 0 GiB, else pause),
  measured once at that context; every placement of a run counts other programs' memory as it
  was when the services started
- the voice worker gates every LLM use: on switching new work waits (at most 180 s), the first
  spoken request that waits hears the notice once under its own id, on ready all go on, on
  paused nothing is answered
- dashboard: "성능 단계: 게임 중 · 게임에 X GB" / 일시정지; game mode screen: game VRAM, revert
  wait, pause in game

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:23:21 +09:00
javis-bot
56212c06dc fix(brain): botBrowser keeps a tab per person, one deadline, and never repeats an action
- each asker has a tab and connection of their own (at most 5, least recently used closes first)
- links, forms and window.open that would open a new tab load in the same tab
- one 15 s deadline covers connecting, the action and the page; events do not extend it
- a broken connection retries only open and read; click, type and back may have run and are not repeated
- <<< and >>> in page content are replaced so it cannot close the fence
- the description says how to search on a site (open, then type with submit=true); eval case 0/3 -> 3/3 on qwen3:8b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:33:00 +09:00
javis-bot
077db5031c feat(brain): botBrowser tool drives the bot's own Chrome over CDP
open, read, click (label or [n]), type (optionally Enter) and back; page content comes back fenced
as untrusted data; only http(s) addresses; no LLM call. Offered only when the app started the bot
browser (JARVIS_BROWSER_CDP) and the browser feature is on; meanwhile fetchWebPage, which has no
address guard, is not offered. Eval evals/test_bot_browser_tool.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:18:05 +09:00
javis-bot
55d19364a0 fix(app): the proxy refuses IPv6 outside global unicast and its stop closes open tunnels
- IPv6 counts as public only in 2000::/3 (without 2001::/23, documentation and 6to4), so site
  local fec0::/10 and IPv4-compatible addresses are refused; the collector uses the same rule
- every proxy connection is a task of one set: a stop ends the open CONNECT tunnels and waits
  until their sockets are gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:07:50 +09:00
javis-bot
69f1bf4fa9 fix(app): collection fetches addresses with Korean paths and words
The request line is ASCII: other characters and spaces in the path and query go as UTF-8 %HH,
while existing %HH and separators stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 21:49:01 +09:00
javis-bot
d0cb540fa0 fix(app): collection's 15 s limit also covers the name lookup, TLS and headers
The name lookup runs on a daemon thread waited for only until the deadline, and a watchdog
shuts the connection down at the deadline, so a server that sends its status line or headers
byte by byte within the per-read limit is cut off too. A response cut short is a failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 21:46:10 +09:00
javis-bot
9185265010 fix(app): collection connects only to the addresses it checked, within 15 s, logging no address
- the fetcher resolves once, requires every address to be public and connects only to those
  (TLS still checks the name), so DNS rebinding cannot reach the local network; redirects too
- 15 s for the whole fetch also against servers that send slowly; no URL in any log line
- a site whose title alone changed is kept again
- target names that differ only in case are refused (one Windows folder)
- the worker always gets the collection file, so results older than the keep days go also
  with no targets or the feature off
- the results screen lists every target with results, not only those in the newest 100

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 21:39:04 +09:00
javis-bot
b94e9dc112 feat(app): background collection frame for sites, search words and RSS feeds
Targets (none by default), interval and keep days on a new 백그라운드 수집 screen with the
results; the worker collects on its own thread through the web search's address check and
writes only to the data folder's collect/ (logs carry target names and counts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 21:29:49 +09:00
javis-bot
8b3905f1c2 fix(app): MCP discovery never blocks a turn; secrets masked as lines are written; TTS child gets no MCP secrets
- Discovery runs outside the MCP tool cache's lock (one at a time), so a turn reading the cached
  tools never waits for a slow server; a new conversation refreshes in the background.
- The log hub masks secrets before a line is kept or written, so a key replaced or removed later
  stays hidden on the screen, in copies, files and exports, also after a restart.
- The worker takes JARVIS_MCP_ENV out of its environment at start, before MeloTTS runs, and hands
  it to the brain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:07:52 +09:00