Commit Graph

159 Commits

Author SHA1 Message Date
javis-bot
716b5bb2e6 feat(app): Jarvis is called like a secretary by default
The default wake words are 비서, 비서야, 비서님, 김비서, 김비서야 and 김비서님.
Where several wake words are heard starting at the same place, the closest
and then the longest is the call, so a call form is removed whole (비서야 멈춰
is a stop) and a near match of a longer word does not eat into what follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:56:38 +09:00
javis-bot
4906d8e7dd chore(app): version 0.1.45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:49:25 +09:00
javis-bot
844958d311 feat(v2): screenView sees someone's Discord screen share in its popped-out window
The share target checks that the person asked about shares their screen in
Jarvis's voice channel, finds Discord's popped-out window (title not ending in
Discord, the voice channel's name first) and, when there is none, asks for it in
the channel and looks again every second for 30 s. Screen actions now wait as
long as PC actions in the worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:43:31 +09:00
javis-bot
3af881935c chore(app): version 0.1.44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
javis-bot
20bd90f65e fix(v2): pcControl keys to a minimised window bring back the window that was in front
restore activates the window (SW_RESTORE), so restoring before keys made the
window already in front: the window that was in front before was not brought
back, and the game / full-screen check came after the window had moved. Keys
now note the window in front and check for a game first, then restore inside
the moment in front. A minimised window never takes keys behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
javis-bot
9a296a5683 chore(app): version 0.1.43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:55:55 +09:00
javis-bot
5767ffa5ae fix(v2): pcControl keys reach a Store app agent-desktop cannot name by its window
On the game PC agent-desktop's press --window-id answers APP_NOT_FOUND for
the new Calculator ("not found with exact process identity", nothing sent),
though focus-window brings it in front. The keys now go to the window in
front while it is still that window. The Windows test checks Notepad's typed
text by element values only (a label such as Ctrl+Shift+X contains "hi").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:55:19 +09:00
javis-bot
378ccfafad feat(v2): pcControl keys go behind a window when they can, else in front for a moment
User decision 2026-11-19 (mixed way). agent-desktop only presses keys in the
window in front, so the app now picks one way before any key is sent:

- behind: one key without modifiers that types or moves is posted to the
  window's one shown text control (Edit, RichEdit, RichEditD2DPT, Scintilla);
  the person's screen and focus stay. Windows does not tell a background
  thread's focused control (win-ci VM: GetFocus is null even attached), so
  only windows with exactly one text control get keys behind.
- in front for a moment: combinations, other keys and every other window;
  the window that was in front is brought back after (also after a failed
  press). A window already in front gets the keys there.
- never in front while game mode is on or the window in front is full screen.
- never both ways, so a key is never pressed twice.

Specs: admin.spec.md "PC control" (Keys), pc-features.md, pc_control.spec.md.
Tests: six new fake-desktop tests in pc.rs; tests/pc_keys_windows.rs on a real
desktop (win-ci VM: letters behind Notepad with Character Map kept in front,
ctrl+a in front and back, the classic Calculator in front and back).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:41:05 +09:00
javis-bot
999c432216 chore(app): version 0.1.42
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 01:04:08 +09:00
javis-bot
912b3304ac chore(app): version 0.1.41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:47:51 +09:00
javis-bot
decd7ae477 fix(v2): YouTube no longer stops 45 s in, consent and the bot's video keep their tools
From the live conversation on 0.1.40 (2026-11-18):

- The bot browser told pages a program drives it (navigator.webdriver), and
  YouTube stopped every video about 45 s in with "문제가 발생했습니다"
  (ump.spsrejectfailure, reproduced twice on the game PC). Chrome now starts
  with --disable-blink-features=AutomationControlled, as the M0 video check
  already did; the live Chrome test checks webdriver is false.
- "네." to "계산기를 실행해도 될까요?" got webSearch and botBrowser from the
  router, so the consent call was refused and the calculator never started.
  Waiting tools: pcControl stays on offer while a request asked in the
  previous reply waits, botBrowser while it holds the asker's page.
- "소리 반으로 줄여 줘" for the bot's video worked on the person's own Edge
  window. botBrowser gets a volume action (media elements and YouTube's player),
  the page shows its volume, its description names its actions first (the part
  the planner sees), pcControl's says the bot's browser is not among its
  windows, and a turn whose plan has no step while a tool holds something for
  the asker gets the one-step nudge for it.

Evals (qwen3.6 on the game PC): volume 0/3 before, 8/8 after; "네." consent
10/10; pcControl 10/10; botBrowser 16/18 (the NewJeans play case fails 0/6
before this change too). Unit: Python 2542 passed, 13 failed (the same 13 as
before); app 348 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:47:38 +09:00
javis-bot
2a57b2b2f4 chore(app): version 0.1.40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:32:37 +09:00
javis-bot
ba44cc224c fix(app): PC control as the real agent-desktop behaves on Windows
Checked against agent-desktop 1.2.0 on the Windows test VM (normal and
elevated sessions); each fault has a test:

- the state folder is made private to the current user: a data folder
  outside the user profile inherits Authenticated Users: Modify and
  agent-desktop then refused every reading (INTERNAL), which broke all
  of PC control on such PCs
- run with the program already open uses the window agent-desktop names
  instead of waiting for a new one (failed after 5.3 s, now 0.6 s);
  several copies get their own reason
- keys to a window Windows will not bring to the front says so
  (NOT_FRONT) and sends nothing
- close that a save dialog holds up, or several copies, get their own
  reasons instead of a generic failure
- an action that closes its own window answers done with no elements
- fields a reading showed as secure are refused as password fields
  before anything is sent

Verified on the VM: calculator 1 2 + 3 = read back 15, Notepad type,
ctrl+a and a letter, elevated windows refused from a normal session and
normal ones not, the emergency stop killing a running command and
blocking acting while reading works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:32:37 +09:00
javis-bot
2e00104332 chore(app): version 0.1.39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:43:18 +09:00
javis-bot
39ac20682e feat(app): PC control through agent-desktop, with the emergency stop and the anti-cheat block
app/src/pc.rs runs every pc_action from the worker (admin.spec.md, "PC
control"): one agent-desktop 1.2.0 run per command with a 15 s limit and
its state in the data folder, never --force or raw input, typed text
only through stdin. Each action is decided by permissions::decide for
its asker (not registered, never, ask without consent), then refused
during the emergency stop, while an anti-cheat game runs (Vanguard,
EAC, BattlEye, FACEIT, GameGuard, XIGNCODE3 and the person's list), on
an administrator window and in a password field. Results carry the
window and its elements read again (200 at most, cut to 120 chars),
agent-desktop's error codes become Korean reasons, and each action is
recorded as pc.<action> with the program's file name only.

The emergency stop is a global hotkey (pcControl.stopHotkey, default
Ctrl+Shift+F12) on its own message loop; it kills the running command
and blocks everything but reading until pressed again or lifted on the
PC control screen. features.pcControl (on by default, nothing happens
until someone is registered) and the pinned exe hash decide whether
the brain gets the tool. The installer ships agent-desktop.exe from the
pinned release archive, both hashes checked.

Tested with a fake agent-desktop (342 lib tests pass) and cargo check
for x86_64-pc-windows-gnu; not yet run against real agent-desktop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:42:48 +09:00
javis-bot
a59ed43ae1 chore(app): version 0.1.38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:55:42 +09:00
javis-bot
4d83543acf chore(app): version 0.1.37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:42:30 +09:00
javis-bot
f05424e1b2 chore(app): version 0.1.36
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:08:35 +09:00
javis-bot
c816358552 chore(app): version 0.1.35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:32:50 +09:00
javis-bot
49050b18a9 chore(app): version 0.1.34
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:09:12 +09:00
javis-bot
e428aad58a chore(app): version 0.1.33
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 17:25:49 +09:00
javis-bot
a665dd8d21 revert(voice): back to upstream songbird 0.6.1, nothing was heard with the held-packet patch
With 0.1.32 (songbird vendored with packets of an unnamed SSRC held until
Discord's Speaking op names it) the bot heard nobody on the live server
from its 16:57 join on 2026-11-18, while 0.1.30 heard every request at
15:58 the same day, and the speaker's Discord was heard by everyone else.
The patch is the only receive change between the two, and the cause was
not found (it passed the two-bot test server runs), so it is removed and
the first ~100 ms of a speaker's first utterance after a join is lost
again, as in 0.1.30.

This reverts 81d1e26 and 9199ccb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 17:24:58 +09:00
javis-bot
a5afdbb221 chore(app): version 0.1.32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:22:30 +09:00
javis-bot
9199ccbd0e fix(voice): release held packets within 20 ms once the speaker is named
Held packets were only processed before the same SSRC's next packet, so
a short first utterance that ended before Discord's Speaking op came was
dropped after a second. The UDP task now also releases every newly named
SSRC's held packets on its 20 ms playout tick.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:22:30 +09:00
javis-bot
2a95f58a0e chore(app): version 0.1.31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:15:23 +09:00
javis-bot
81d1e26f61 fix(voice): keep the first packets of a speaker until Discord names them
Discord names a speaker's SSRC in the Speaking op only the first time
they speak, and the op arrives after their first UDP packets. songbird
0.6.1 dropped the DAVE packets it could not decrypt yet, so the first
~100 ms of the first thing each person said after a join were lost
(live 2026-11-18: "자비스 서울 날씨 어때" arrived as "아비 서울날씨
어때", and the first call after the 01:20 start as "아비스").

songbird is vendored in app/vendor/songbird with one patch: packets of
an unnamed SSRC are held (1 s, 50 packets each, 32 SSRCs) and processed
in arrival order once it is named, and the Speaking event is sent before
the SSRC is named so handlers know the speaker first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:15:07 +09:00
javis-bot
2e90c47a95 chore(app): version 0.1.30
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:21:46 +09:00
javis-bot
b4f8f71dd2 chore(app): version 0.1.29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:08:15 +09:00
javis-bot
1bef5db912 chore(app): version 0.1.28
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:57:23 +09:00
javis-bot
17927b6450 feat(weather): KMA forecasts for a later day (days_ahead, days)
getWeather takes days_ahead (days from today, up to 10) and days (a span,
2 for a weekend). Inside Korea with the app's KMA key: 단기예보 for today
to 3 days ahead (morning and afternoon weather, lowest and highest, highest
chance of rain), then 중기기온 and 중기육상 for the nearest of 32 KMA cities
and its region. Days KMA does not give, and every KMA failure, come from
open-meteo's daily forecast. days_ahead 0 with more days gives the weather
now, then each day. Eval: evals/test_weather_forecast_days.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:57:12 +09:00
javis-bot
ecde1e6bd0 chore(app): version 0.1.27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:45 +09:00
javis-bot
282f114e60 feat(weather): KMA current weather inside Korea with the app's stored key
The 기능 screen's 기상청 날씨 card stores the KMA API hub key encrypted
(kmaApiKey, masked in logs). On start the app passes it to the worker in
JARVIS_KMA_API_KEY; the worker takes it out of its environment before any
child process starts. Inside Korea getWeather reads 초단기실황 and the
coming hour's sky from 초단기예보, and falls back to open-meteo, then
met.no, on any KMA failure. The key is in KMA's URLs, so failures are
logged by their kind only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:31 +09:00
javis-bot
68d0333698 chore(app): version 0.1.26
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:24:11 +09:00
javis-bot
1c06b16bd4 chore(app): version 0.1.25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
44387685af fix(app): log the card's real free memory, and short answers get their line too
Free memory is nvidia-smi's memory.free in the app and the voice worker, so the
driver's reserved share no longer counts as free. A card reading still running
when an answer ends finishes on its own and then logs, so an answer shorter
than one reading is logged too, and the answer still never waits for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
61b68130c6 chore(app): version 0.1.24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:48 +09:00
javis-bot
f1600f09c2 feat(app): log the graphics card's free memory after the LLM starts and while each answer is written
The app logs how much of the card is free once the model answers, and the
voice worker reads the card once a second while an answer or a scheduled task
is written and logs the least free memory seen (no text, llm tab), so a slow
answer can be matched to a full card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:20 +09:00
javis-bot
ce03424ba4 docs(spec): PC control tool, next-message consent, agent-desktop in the app
pcControl (v2 M7): windows, read, click, type, keys, run, close through the
app's agent-desktop. Ask-level actions wait for the asker's own next message
and then cover that window's asked kinds for the reply; payments, passwords,
deleting files and admin rights are filtered before asking and never done.
The app refuses during an anti-cheat game and the emergency stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 14:04:43 +09:00
javis-bot
d02092662b chore(app): version 0.1.23
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:07:48 +09:00
javis-bot
4adc6a8d72 fix(app): resident launcher apps are not games, and the TTS keeps its share of the card
On the game PC, Wallpaper Engine (steamapps\common\wallpaper_engine) and the
Riot client's crash handler (Riot Games\Riot Client) were taken for games
running behind other windows as the services started, so no reading of other
programs was kept and the game stage placed 0 GiB of experts instead of 2.

- A launcher's own folders and non-game apps under its game folder are not
  game folders (Steamworks Shared, wallpaper_engine; Launcher, Epic Online
  Services; Riot Client).
- The game log tab names the program found running at the start.
- The placement keeps the TTS's share (MeloTTS, 1005 MiB, M0 tts_melo) beside
  the STT's, and takes the worker's loaded models off what the card holds, so
  a start (TTS not yet loaded) and a switch (TTS loaded) place the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:07:45 +09:00
javis-bot
4993942040 chore(app): version 0.1.22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 10:11:59 +09:00
javis-bot
ff78c0ac41 feat(app): the LLM always takes one request at a time, and only the latest call is answered by default
A chat channel answering every message no longer starts Strata with
--parallel 2: on the 16 GB card that read prompts about three times
slower (2026-11-16, answers took 10-20 s). Chat and voice answers now
take turns at Strata. response.policy defaults to latest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 10:01:37 +09:00
javis-bot
07615ee7a1 chore(app): version 0.1.21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 21:13:39 +09:00
javis-bot
6a9b66bdcd chore(app): version 0.1.20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:26:56 +09:00
javis-bot
cda8db7fce fix: play on YouTube in one step, Korea as the default location, searches in the user's language
- botBrowser `play`: searches YouTube for the words as said, opens the first video (not an ad,
  without the playlist) and starts it, in one call (an address given opens that page); `type`
  also takes `query`. Live 0.1.19: the model stopped on YouTube's home page or the results, a
  retry was refused as a third call, and the answer still said the video played.
- `brain.location` (기본 위치, default 대한민국) goes to the brain as `location_name`: the time line
  says it without GeoIP (no database on the game PC), and getWeather without a place uses it.
- Planner and webSearch keep search words in the user's language (never translated into
  English), so recommendations fit where the user is.

Evals (qwen3:8b, 3 runs): play ending on a playing video found with the words as said 0/6 -> 5/6;
Korean searches for Korean requests 6/9 -> 8/9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:26:30 +09:00
javis-bot
65294f3f97 chore(app): version 0.1.19
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:12:20 +09:00
javis-bot
420c9c73f3 fix: stopped sentences recorded only as far as heard, search words as plain query, paused video said so
- Playback sends how long the clip playing when a request was stopped had played (tts_played
  last_ms); the worker cuts the last sentence of that recording to it.
- botBrowser open takes the search words as plain `query`: the tool puts them, encoded, into the
  address's empty search parameter or types them into the site's search box. The description
  tells the model to search this way and never put words into an address itself (live: 아이유
  hand-encoded into 아이우). Eval (qwen3:8b, 3 runs): right words reach YouTube 1/3 -> 3/3.
- When a page's main video does not play (also after the tool pressed play, or it failed to load),
  a note after the page says it is not playing and not to say it plays. Eval: a paused video
  reported as playing 1/3 -> 0/3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:12:04 +09:00
javis-bot
9a91bc9ab6 chore(app): version 0.1.18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:56:40 +09:00
javis-bot
ec2672cd4a fix: recordings keep only what was played, newest waiting request kept, video pages opened play
- Playback tells the worker how much of each request was played (tts_played); the worker keeps a
  spoken answer's recording only that far (an answer stopped mid-way no longer keeps its unheard
  rest, and is marked stopped); nothing played or never reported: no file.
- Old recording day folders are also removed with the first recording of each new day.
- response.interrupt off, latest: a request whose final comes after a newer one's is replaced
  instead of replacing it.
- botBrowser presses play on a page's paused main player also after open, not only after a click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:56:40 +09:00
javis-bot
fdcaae4820 chore(app): version 0.1.17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:43:10 +09:00