Commit Graph

166 Commits

Author SHA1 Message Date
javis-bot
09bdf05762 chore(app): version 0.1.49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 20:30:41 +09:00
javis-bot
251f5e116d fix(conversation,voice_worker): going on survives the answer starting, ignores empty finals; cancel checks before every model call; context finals stay batched
- A speaker who starts their next utterance within call.continueMs holds their request open for
  that final, also if the answer starts playing meanwhile; an empty final (rejected noise) joins
  nothing and asks nothing again
- The engine checks the cancel before the tool router, the planner and a planned tool call too
- Finals with a context are batched again: each context is cut from its end to the shortest one in
  the batch (ctranslate2 needs the start token at one position); ten finals at once 1.3 s
- Measured harsh conditions (docs/v2/stt-accuracy-plan.md): the context never made them worse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 20:30:29 +09:00
javis-bot
9a527aa470 chore(app): version 0.1.48
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:41:46 +09:00
javis-bot
478968949c chore(app): version 0.1.47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:42:02 +09:00
javis-bot
caad09c613 feat(conversation): what is said in pieces is answered once, and a dropped answer stops the brain
Live 2026-11-19 every piece of one thought (pauses of 0.1-4.7 s) was answered on its own and the
brain wrote each cancelled answer to its end, so the replies queued up to 35 s.

- conversation: a speaker who goes on before their answer is heard (within call.continueMs,
  default 5 s) has the earlier request taken back (policy withdraw, not missed) and the joined
  text answered as one request; a stop still stops; no second earcon; partials prepare the joined
  text
- voice worker: a reply cancelled while waiting is never started, a running one stops at the
  engine's next model or tool call (turn_state.cancel), and it uses up no turn, so a confirmation
  carried by the joined request does not lapse
- specs: conversation.spec.md (Going on), voice_worker.spec.md, llm_contexts.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:41:50 +09:00
javis-bot
e32671f871 chore(app): version 0.1.46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:01:40 +09:00
javis-bot
e23a962923 feat(app): side menu in groups that open, settings split into tidy screens
The side menu keeps 대시보드 and 대화 on top and puts the rest in groups that
open when their name is pressed (디스코드, 대화 방식, 음성·모델, 기능, PC, 관리);
the shown screen's group opens and open groups are kept. 채널·호출 is split
into 음성 채널, 호출 and 여럿이 말할 때, and 모델 into 받아 적기·말하기 and 대화
모델; the chat wake words join 호출, the recording switch 음성 채널 and the
default location 기능 켜기·MCP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:01:16 +09:00
javis-bot
716b5bb2e6 feat(app): Jarvis is called like a secretary by default
The default wake words are 비서, 비서야, 비서님, 김비서, 김비서야 and 김비서님.
Where several wake words are heard starting at the same place, the closest
and then the longest is the call, so a call form is removed whole (비서야 멈춰
is a stop) and a near match of a longer word does not eat into what follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:56:38 +09:00
javis-bot
4906d8e7dd chore(app): version 0.1.45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:49:25 +09:00
javis-bot
844958d311 feat(v2): screenView sees someone's Discord screen share in its popped-out window
The share target checks that the person asked about shares their screen in
Jarvis's voice channel, finds Discord's popped-out window (title not ending in
Discord, the voice channel's name first) and, when there is none, asks for it in
the channel and looks again every second for 30 s. Screen actions now wait as
long as PC actions in the worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:43:31 +09:00
javis-bot
3af881935c chore(app): version 0.1.44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
javis-bot
20bd90f65e fix(v2): pcControl keys to a minimised window bring back the window that was in front
restore activates the window (SW_RESTORE), so restoring before keys made the
window already in front: the window that was in front before was not brought
back, and the game / full-screen check came after the window had moved. Keys
now note the window in front and check for a game first, then restore inside
the moment in front. A minimised window never takes keys behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
javis-bot
9a296a5683 chore(app): version 0.1.43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:55:55 +09:00
javis-bot
5767ffa5ae fix(v2): pcControl keys reach a Store app agent-desktop cannot name by its window
On the game PC agent-desktop's press --window-id answers APP_NOT_FOUND for
the new Calculator ("not found with exact process identity", nothing sent),
though focus-window brings it in front. The keys now go to the window in
front while it is still that window. The Windows test checks Notepad's typed
text by element values only (a label such as Ctrl+Shift+X contains "hi").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:55:19 +09:00
javis-bot
378ccfafad feat(v2): pcControl keys go behind a window when they can, else in front for a moment
User decision 2026-11-19 (mixed way). agent-desktop only presses keys in the
window in front, so the app now picks one way before any key is sent:

- behind: one key without modifiers that types or moves is posted to the
  window's one shown text control (Edit, RichEdit, RichEditD2DPT, Scintilla);
  the person's screen and focus stay. Windows does not tell a background
  thread's focused control (win-ci VM: GetFocus is null even attached), so
  only windows with exactly one text control get keys behind.
- in front for a moment: combinations, other keys and every other window;
  the window that was in front is brought back after (also after a failed
  press). A window already in front gets the keys there.
- never in front while game mode is on or the window in front is full screen.
- never both ways, so a key is never pressed twice.

Specs: admin.spec.md "PC control" (Keys), pc-features.md, pc_control.spec.md.
Tests: six new fake-desktop tests in pc.rs; tests/pc_keys_windows.rs on a real
desktop (win-ci VM: letters behind Notepad with Character Map kept in front,
ctrl+a in front and back, the classic Calculator in front and back).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:41:05 +09:00
javis-bot
999c432216 chore(app): version 0.1.42
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 01:04:08 +09:00
javis-bot
912b3304ac chore(app): version 0.1.41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:47:51 +09:00
javis-bot
decd7ae477 fix(v2): YouTube no longer stops 45 s in, consent and the bot's video keep their tools
From the live conversation on 0.1.40 (2026-11-18):

- The bot browser told pages a program drives it (navigator.webdriver), and
  YouTube stopped every video about 45 s in with "문제가 발생했습니다"
  (ump.spsrejectfailure, reproduced twice on the game PC). Chrome now starts
  with --disable-blink-features=AutomationControlled, as the M0 video check
  already did; the live Chrome test checks webdriver is false.
- "네." to "계산기를 실행해도 될까요?" got webSearch and botBrowser from the
  router, so the consent call was refused and the calculator never started.
  Waiting tools: pcControl stays on offer while a request asked in the
  previous reply waits, botBrowser while it holds the asker's page.
- "소리 반으로 줄여 줘" for the bot's video worked on the person's own Edge
  window. botBrowser gets a volume action (media elements and YouTube's player),
  the page shows its volume, its description names its actions first (the part
  the planner sees), pcControl's says the bot's browser is not among its
  windows, and a turn whose plan has no step while a tool holds something for
  the asker gets the one-step nudge for it.

Evals (qwen3.6 on the game PC): volume 0/3 before, 8/8 after; "네." consent
10/10; pcControl 10/10; botBrowser 16/18 (the NewJeans play case fails 0/6
before this change too). Unit: Python 2542 passed, 13 failed (the same 13 as
before); app 348 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:47:38 +09:00
javis-bot
2a57b2b2f4 chore(app): version 0.1.40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:32:37 +09:00
javis-bot
ba44cc224c fix(app): PC control as the real agent-desktop behaves on Windows
Checked against agent-desktop 1.2.0 on the Windows test VM (normal and
elevated sessions); each fault has a test:

- the state folder is made private to the current user: a data folder
  outside the user profile inherits Authenticated Users: Modify and
  agent-desktop then refused every reading (INTERNAL), which broke all
  of PC control on such PCs
- run with the program already open uses the window agent-desktop names
  instead of waiting for a new one (failed after 5.3 s, now 0.6 s);
  several copies get their own reason
- keys to a window Windows will not bring to the front says so
  (NOT_FRONT) and sends nothing
- close that a save dialog holds up, or several copies, get their own
  reasons instead of a generic failure
- an action that closes its own window answers done with no elements
- fields a reading showed as secure are refused as password fields
  before anything is sent

Verified on the VM: calculator 1 2 + 3 = read back 15, Notepad type,
ctrl+a and a letter, elevated windows refused from a normal session and
normal ones not, the emergency stop killing a running command and
blocking acting while reading works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:32:37 +09:00
javis-bot
2e00104332 chore(app): version 0.1.39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:43:18 +09:00
javis-bot
39ac20682e feat(app): PC control through agent-desktop, with the emergency stop and the anti-cheat block
app/src/pc.rs runs every pc_action from the worker (admin.spec.md, "PC
control"): one agent-desktop 1.2.0 run per command with a 15 s limit and
its state in the data folder, never --force or raw input, typed text
only through stdin. Each action is decided by permissions::decide for
its asker (not registered, never, ask without consent), then refused
during the emergency stop, while an anti-cheat game runs (Vanguard,
EAC, BattlEye, FACEIT, GameGuard, XIGNCODE3 and the person's list), on
an administrator window and in a password field. Results carry the
window and its elements read again (200 at most, cut to 120 chars),
agent-desktop's error codes become Korean reasons, and each action is
recorded as pc.<action> with the program's file name only.

The emergency stop is a global hotkey (pcControl.stopHotkey, default
Ctrl+Shift+F12) on its own message loop; it kills the running command
and blocks everything but reading until pressed again or lifted on the
PC control screen. features.pcControl (on by default, nothing happens
until someone is registered) and the pinned exe hash decide whether
the brain gets the tool. The installer ships agent-desktop.exe from the
pinned release archive, both hashes checked.

Tested with a fake agent-desktop (342 lib tests pass) and cargo check
for x86_64-pc-windows-gnu; not yet run against real agent-desktop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:42:48 +09:00
javis-bot
a59ed43ae1 chore(app): version 0.1.38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:55:42 +09:00
javis-bot
4d83543acf chore(app): version 0.1.37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:42:30 +09:00
javis-bot
f05424e1b2 chore(app): version 0.1.36
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:08:35 +09:00
javis-bot
c816358552 chore(app): version 0.1.35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:32:50 +09:00
javis-bot
49050b18a9 chore(app): version 0.1.34
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:09:12 +09:00
javis-bot
e428aad58a chore(app): version 0.1.33
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 17:25:49 +09:00
javis-bot
a665dd8d21 revert(voice): back to upstream songbird 0.6.1, nothing was heard with the held-packet patch
With 0.1.32 (songbird vendored with packets of an unnamed SSRC held until
Discord's Speaking op names it) the bot heard nobody on the live server
from its 16:57 join on 2026-11-18, while 0.1.30 heard every request at
15:58 the same day, and the speaker's Discord was heard by everyone else.
The patch is the only receive change between the two, and the cause was
not found (it passed the two-bot test server runs), so it is removed and
the first ~100 ms of a speaker's first utterance after a join is lost
again, as in 0.1.30.

This reverts 81d1e26 and 9199ccb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 17:24:58 +09:00
javis-bot
a5afdbb221 chore(app): version 0.1.32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:22:30 +09:00
javis-bot
9199ccbd0e fix(voice): release held packets within 20 ms once the speaker is named
Held packets were only processed before the same SSRC's next packet, so
a short first utterance that ended before Discord's Speaking op came was
dropped after a second. The UDP task now also releases every newly named
SSRC's held packets on its 20 ms playout tick.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:22:30 +09:00
javis-bot
2a95f58a0e chore(app): version 0.1.31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:15:23 +09:00
javis-bot
81d1e26f61 fix(voice): keep the first packets of a speaker until Discord names them
Discord names a speaker's SSRC in the Speaking op only the first time
they speak, and the op arrives after their first UDP packets. songbird
0.6.1 dropped the DAVE packets it could not decrypt yet, so the first
~100 ms of the first thing each person said after a join were lost
(live 2026-11-18: "자비스 서울 날씨 어때" arrived as "아비 서울날씨
어때", and the first call after the 01:20 start as "아비스").

songbird is vendored in app/vendor/songbird with one patch: packets of
an unnamed SSRC are held (1 s, 50 packets each, 32 SSRCs) and processed
in arrival order once it is named, and the Speaking event is sent before
the SSRC is named so handlers know the speaker first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:15:07 +09:00
javis-bot
2e90c47a95 chore(app): version 0.1.30
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:21:46 +09:00
javis-bot
b4f8f71dd2 chore(app): version 0.1.29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:08:15 +09:00
javis-bot
1bef5db912 chore(app): version 0.1.28
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:57:23 +09:00
javis-bot
17927b6450 feat(weather): KMA forecasts for a later day (days_ahead, days)
getWeather takes days_ahead (days from today, up to 10) and days (a span,
2 for a weekend). Inside Korea with the app's KMA key: 단기예보 for today
to 3 days ahead (morning and afternoon weather, lowest and highest, highest
chance of rain), then 중기기온 and 중기육상 for the nearest of 32 KMA cities
and its region. Days KMA does not give, and every KMA failure, come from
open-meteo's daily forecast. days_ahead 0 with more days gives the weather
now, then each day. Eval: evals/test_weather_forecast_days.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:57:12 +09:00
javis-bot
ecde1e6bd0 chore(app): version 0.1.27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:45 +09:00
javis-bot
282f114e60 feat(weather): KMA current weather inside Korea with the app's stored key
The 기능 screen's 기상청 날씨 card stores the KMA API hub key encrypted
(kmaApiKey, masked in logs). On start the app passes it to the worker in
JARVIS_KMA_API_KEY; the worker takes it out of its environment before any
child process starts. Inside Korea getWeather reads 초단기실황 and the
coming hour's sky from 초단기예보, and falls back to open-meteo, then
met.no, on any KMA failure. The key is in KMA's URLs, so failures are
logged by their kind only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:31 +09:00
javis-bot
68d0333698 chore(app): version 0.1.26
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:24:11 +09:00
javis-bot
1c06b16bd4 chore(app): version 0.1.25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
44387685af fix(app): log the card's real free memory, and short answers get their line too
Free memory is nvidia-smi's memory.free in the app and the voice worker, so the
driver's reserved share no longer counts as free. A card reading still running
when an answer ends finishes on its own and then logs, so an answer shorter
than one reading is logged too, and the answer still never waits for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:48:01 +09:00
javis-bot
61b68130c6 chore(app): version 0.1.24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:48 +09:00
javis-bot
f1600f09c2 feat(app): log the graphics card's free memory after the LLM starts and while each answer is written
The app logs how much of the card is free once the model answers, and the
voice worker reads the card once a second while an answer or a scheduled task
is written and logs the least free memory seen (no text, llm tab), so a slow
answer can be matched to a full card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:40:20 +09:00
javis-bot
ce03424ba4 docs(spec): PC control tool, next-message consent, agent-desktop in the app
pcControl (v2 M7): windows, read, click, type, keys, run, close through the
app's agent-desktop. Ask-level actions wait for the asker's own next message
and then cover that window's asked kinds for the reply; payments, passwords,
deleting files and admin rights are filtered before asking and never done.
The app refuses during an anti-cheat game and the emergency stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 14:04:43 +09:00
javis-bot
d02092662b chore(app): version 0.1.23
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:07:48 +09:00
javis-bot
4adc6a8d72 fix(app): resident launcher apps are not games, and the TTS keeps its share of the card
On the game PC, Wallpaper Engine (steamapps\common\wallpaper_engine) and the
Riot client's crash handler (Riot Games\Riot Client) were taken for games
running behind other windows as the services started, so no reading of other
programs was kept and the game stage placed 0 GiB of experts instead of 2.

- A launcher's own folders and non-game apps under its game folder are not
  game folders (Steamworks Shared, wallpaper_engine; Launcher, Epic Online
  Services; Riot Client).
- The game log tab names the program found running at the start.
- The placement keeps the TTS's share (MeloTTS, 1005 MiB, M0 tts_melo) beside
  the STT's, and takes the worker's loaded models off what the card holds, so
  a start (TTS not yet loaded) and a switch (TTS loaded) place the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:07:45 +09:00
javis-bot
4993942040 chore(app): version 0.1.22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 10:11:59 +09:00
javis-bot
ff78c0ac41 feat(app): the LLM always takes one request at a time, and only the latest call is answered by default
A chat channel answering every message no longer starts Strata with
--parallel 2: on the 16 GB card that read prompts about three times
slower (2026-11-16, answers took 10-20 s). Chat and voice answers now
take turns at Strata. response.policy defaults to latest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 10:01:37 +09:00
javis-bot
07615ee7a1 chore(app): version 0.1.21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 21:13:39 +09:00