On the game PC agent-desktop's press --window-id answers APP_NOT_FOUND for
the new Calculator ("not found with exact process identity", nothing sent),
though focus-window brings it in front. The keys now go to the window in
front while it is still that window. The Windows test checks Notepad's typed
text by element values only (a label such as Ctrl+Shift+X contains "hi").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User decision 2026-11-19 (mixed way). agent-desktop only presses keys in the
window in front, so the app now picks one way before any key is sent:
- behind: one key without modifiers that types or moves is posted to the
window's one shown text control (Edit, RichEdit, RichEditD2DPT, Scintilla);
the person's screen and focus stay. Windows does not tell a background
thread's focused control (win-ci VM: GetFocus is null even attached), so
only windows with exactly one text control get keys behind.
- in front for a moment: combinations, other keys and every other window;
the window that was in front is brought back after (also after a failed
press). A window already in front gets the keys there.
- never in front while game mode is on or the window in front is full screen.
- never both ways, so a key is never pressed twice.
Specs: admin.spec.md "PC control" (Keys), pc-features.md, pc_control.spec.md.
Tests: six new fake-desktop tests in pc.rs; tests/pc_keys_windows.rs on a real
desktop (win-ci VM: letters behind Notepad with Character Map kept in front,
ctrl+a in front and back, the classic Calculator in front and back).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the live conversation on 0.1.40 (2026-11-18):
- The bot browser told pages a program drives it (navigator.webdriver), and
YouTube stopped every video about 45 s in with "문제가 발생했습니다"
(ump.spsrejectfailure, reproduced twice on the game PC). Chrome now starts
with --disable-blink-features=AutomationControlled, as the M0 video check
already did; the live Chrome test checks webdriver is false.
- "네." to "계산기를 실행해도 될까요?" got webSearch and botBrowser from the
router, so the consent call was refused and the calculator never started.
Waiting tools: pcControl stays on offer while a request asked in the
previous reply waits, botBrowser while it holds the asker's page.
- "소리 반으로 줄여 줘" for the bot's video worked on the person's own Edge
window. botBrowser gets a volume action (media elements and YouTube's player),
the page shows its volume, its description names its actions first (the part
the planner sees), pcControl's says the bot's browser is not among its
windows, and a turn whose plan has no step while a tool holds something for
the asker gets the one-step nudge for it.
Evals (qwen3.6 on the game PC): volume 0/3 before, 8/8 after; "네." consent
10/10; pcControl 10/10; botBrowser 16/18 (the NewJeans play case fails 0/6
before this change too). Unit: Python 2542 passed, 13 failed (the same 13 as
before); app 348 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checked against agent-desktop 1.2.0 on the Windows test VM (normal and
elevated sessions); each fault has a test:
- the state folder is made private to the current user: a data folder
outside the user profile inherits Authenticated Users: Modify and
agent-desktop then refused every reading (INTERNAL), which broke all
of PC control on such PCs
- run with the program already open uses the window agent-desktop names
instead of waiting for a new one (failed after 5.3 s, now 0.6 s);
several copies get their own reason
- keys to a window Windows will not bring to the front says so
(NOT_FRONT) and sends nothing
- close that a save dialog holds up, or several copies, get their own
reasons instead of a generic failure
- an action that closes its own window answers done with no elements
- fields a reading showed as secure are refused as password fields
before anything is sent
Verified on the VM: calculator 1 2 + 3 = read back 15, Notepad type,
ctrl+a and a letter, elevated windows refused from a normal session and
normal ones not, the emergency stop killing a running command and
blocking acting while reading works.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
app/src/pc.rs runs every pc_action from the worker (admin.spec.md, "PC
control"): one agent-desktop 1.2.0 run per command with a 15 s limit and
its state in the data folder, never --force or raw input, typed text
only through stdin. Each action is decided by permissions::decide for
its asker (not registered, never, ask without consent), then refused
during the emergency stop, while an anti-cheat game runs (Vanguard,
EAC, BattlEye, FACEIT, GameGuard, XIGNCODE3 and the person's list), on
an administrator window and in a password field. Results carry the
window and its elements read again (200 at most, cut to 120 chars),
agent-desktop's error codes become Korean reasons, and each action is
recorded as pc.<action> with the program's file name only.
The emergency stop is a global hotkey (pcControl.stopHotkey, default
Ctrl+Shift+F12) on its own message loop; it kills the running command
and blocks everything but reading until pressed again or lifted on the
PC control screen. features.pcControl (on by default, nothing happens
until someone is registered) and the pinned exe hash decide whether
the brain gets the tool. The installer ships agent-desktop.exe from the
pinned release archive, both hashes checked.
Tested with a fake agent-desktop (342 lib tests pass) and cargo check
for x86_64-pc-windows-gnu; not yet run against real agent-desktop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With 0.1.32 (songbird vendored with packets of an unnamed SSRC held until
Discord's Speaking op names it) the bot heard nobody on the live server
from its 16:57 join on 2026-11-18, while 0.1.30 heard every request at
15:58 the same day, and the speaker's Discord was heard by everyone else.
The patch is the only receive change between the two, and the cause was
not found (it passed the two-bot test server runs), so it is removed and
the first ~100 ms of a speaker's first utterance after a join is lost
again, as in 0.1.30.
This reverts 81d1e26 and 9199ccb.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Held packets were only processed before the same SSRC's next packet, so
a short first utterance that ended before Discord's Speaking op came was
dropped after a second. The UDP task now also releases every newly named
SSRC's held packets on its 20 ms playout tick.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Discord names a speaker's SSRC in the Speaking op only the first time
they speak, and the op arrives after their first UDP packets. songbird
0.6.1 dropped the DAVE packets it could not decrypt yet, so the first
~100 ms of the first thing each person said after a join were lost
(live 2026-11-18: "자비스 서울 날씨 어때" arrived as "아비 서울날씨
어때", and the first call after the 01:20 start as "아비스").
songbird is vendored in app/vendor/songbird with one patch: packets of
an unnamed SSRC are held (1 s, 50 packets each, 32 SSRCs) and processed
in arrival order once it is named, and the Speaking event is sent before
the SSRC is named so handlers know the speaker first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getWeather takes days_ahead (days from today, up to 10) and days (a span,
2 for a weekend). Inside Korea with the app's KMA key: 단기예보 for today
to 3 days ahead (morning and afternoon weather, lowest and highest, highest
chance of rain), then 중기기온 and 중기육상 for the nearest of 32 KMA cities
and its region. Days KMA does not give, and every KMA failure, come from
open-meteo's daily forecast. days_ahead 0 with more days gives the weather
now, then each day. Eval: evals/test_weather_forecast_days.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 기능 screen's 기상청 날씨 card stores the KMA API hub key encrypted
(kmaApiKey, masked in logs). On start the app passes it to the worker in
JARVIS_KMA_API_KEY; the worker takes it out of its environment before any
child process starts. Inside Korea getWeather reads 초단기실황 and the
coming hour's sky from 초단기예보, and falls back to open-meteo, then
met.no, on any KMA failure. The key is in KMA's URLs, so failures are
logged by their kind only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Free memory is nvidia-smi's memory.free in the app and the voice worker, so the
driver's reserved share no longer counts as free. A card reading still running
when an answer ends finishes on its own and then logs, so an answer shorter
than one reading is logged too, and the answer still never waits for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The app logs how much of the card is free once the model answers, and the
voice worker reads the card once a second while an answer or a scheduled task
is written and logs the least free memory seen (no text, llm tab), so a slow
answer can be matched to a full card.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pcControl (v2 M7): windows, read, click, type, keys, run, close through the
app's agent-desktop. Ask-level actions wait for the asker's own next message
and then cover that window's asked kinds for the reply; payments, passwords,
deleting files and admin rights are filtered before asking and never done.
The app refuses during an anti-cheat game and the emergency stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the game PC, Wallpaper Engine (steamapps\common\wallpaper_engine) and the
Riot client's crash handler (Riot Games\Riot Client) were taken for games
running behind other windows as the services started, so no reading of other
programs was kept and the game stage placed 0 GiB of experts instead of 2.
- A launcher's own folders and non-game apps under its game folder are not
game folders (Steamworks Shared, wallpaper_engine; Launcher, Epic Online
Services; Riot Client).
- The game log tab names the program found running at the start.
- The placement keeps the TTS's share (MeloTTS, 1005 MiB, M0 tts_melo) beside
the STT's, and takes the worker's loaded models off what the card holds, so
a start (TTS not yet loaded) and a switch (TTS loaded) place the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chat channel answering every message no longer starts Strata with
--parallel 2: on the 16 GB card that read prompts about three times
slower (2026-11-16, answers took 10-20 s). Chat and voice answers now
take turns at Strata. response.policy defaults to latest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- botBrowser `play`: searches YouTube for the words as said, opens the first video (not an ad,
without the playlist) and starts it, in one call (an address given opens that page); `type`
also takes `query`. Live 0.1.19: the model stopped on YouTube's home page or the results, a
retry was refused as a third call, and the answer still said the video played.
- `brain.location` (기본 위치, default 대한민국) goes to the brain as `location_name`: the time line
says it without GeoIP (no database on the game PC), and getWeather without a place uses it.
- Planner and webSearch keep search words in the user's language (never translated into
English), so recommendations fit where the user is.
Evals (qwen3:8b, 3 runs): play ending on a playing video found with the words as said 0/6 -> 5/6;
Korean searches for Korean requests 6/9 -> 8/9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Playback sends how long the clip playing when a request was stopped had played (tts_played
last_ms); the worker cuts the last sentence of that recording to it.
- botBrowser open takes the search words as plain `query`: the tool puts them, encoded, into the
address's empty search parameter or types them into the site's search box. The description
tells the model to search this way and never put words into an address itself (live: 아이유
hand-encoded into 아이우). Eval (qwen3:8b, 3 runs): right words reach YouTube 1/3 -> 3/3.
- When a page's main video does not play (also after the tool pressed play, or it failed to load),
a note after the page says it is not playing and not to say it plays. Eval: a paused video
reported as playing 1/3 -> 0/3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Playback tells the worker how much of each request was played (tts_played); the worker keeps a
spoken answer's recording only that far (an answer stopped mid-way no longer keeps its unheard
rest, and is marked stopped); nothing played or never reported: no file.
- Old recording day folders are also removed with the first recording of each new day.
- response.interrupt off, latest: a request whose final comes after a newer one's is replaced
instead of replacing it.
- botBrowser presses play on a page's paused main player also after open, not only after a click.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alone with Jarvis, whatever the speaker says is answered without asking the judge. With others
present the judge is told how many people are in the channel and how many may call Jarvis: with
one other person most of what the speaker says counts as said to Jarvis (unclear is yes); with
more, or not known, the question is the group one it was before. Live 2026-11-16: alone in the
channel, questions without the wake word got no answer because the judge assumed a group.
Eval (qwen3:8b, pair cases): 3/7 before, 6/7 after; group cases unchanged.
Also: the protocol test fixture's brain takes the trace argument (the test failed since traces).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- 로그 screen: "보이는 글 txt로 저장" (logs_save_text), a 대화 view that saves the days'
conversation with every trace in full (history_export), and a 목소리 view with the
voice.record switch and the days' recordings as a zip with an index (voice_export).
- Voice worker: with JARVIS_RECORDINGS_DIR (voice.record, off by default) each final and each
spoken answer is kept as a WAV with an index line; day folders older than 7 days removed.
- response.interrupt (default on): off, latest/important finish the running answer first.
- Bot browser: Chrome lets pages start video and sound; botBrowser reports each page's media
state, waits for a player to start after open/click, and presses play on a video page a click
led to when its player stayed paused (live: a clicked YouTube video stayed paused).
- webSearch reads `query` when `search_query` is missing (live: both searches of an answer failed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The screen re-read its window every 3 seconds unless a trace was already
shown as being made, so an answer that ended between two reads came with
its trace. It now asks twice a second for history_stamp (the newest
record and trace ids and the steps of the traces being made, cheap and not
logged) and reads the window again as soon as it changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>