With the 3 s wait a short answer can play and end inside the speaker's pause (answer
3-4 s, next piece at 4.7 s): the request was dropped at SpeakEnd and the next piece became
a new request. A request now stays open for call.continueMs after it ended whatever its
answer does; a SpeechStart after the window drops it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live 2026-11-19 (the user's own pieced requests replayed into Discord) the pauses inside
one thought, 2.3-4.7 s, were longer than writing the answer: the answer started in the
pause, and what followed became a new request, so answers were cut off mid-sentence.
- A voice answer is written at once but starts playing only after its asker has been
quiet for call.answerWaitMs (default 3000, 0 = off, settings "말이 끝나고 답하기 전
기다림"); while they talk it waits for their final, which joins the request.
- A request stays open (Going on) until its answer ended, not until it is first heard:
going on within call.continueMs while it plays stops it and answers all of it.
Replayed live: the three pieced requests each get one answer, none cut off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live 2026-11-19 "비서야" was decoded as "기사야" (two of six letters off, more than call
detection allows) while the hinted decode of the same audio gave "비서야"; the hint was
dropped because it changed the wake word count. The hint may now add one wake word where
the plain text is within a third of that word's letters (NFKD letters, Hangul by jamo).
On the user's 100 recordings: 5 fixed, 0 wrong; 0 of 216 noise decodes gained a wake word.
The live round trip test gets a going-on case: the person's own recordings of one thing
said in pieces, played into Discord with their real pauses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Harsh copies of the user's speech (weak voice, noise at SNR 10/5 dB, headset band, Opus 12 kbps,
reverb) with the worker's own VAD for noise: lowering the gate itself let laughter through (VAD-cut
noise 0/5 -> 1/5 at -0.8), and three noise suppressors (DeepFilterNet3, RNNoise, noisereduce) made
every condition worse, so neither is used. Rescuing a plain decode at -0.8..-0.7 with the prompted
text when that passed the gate and stays close: headset 13.8% -> 7.5%, no more noise with text.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A speaker who starts their next utterance within call.continueMs holds their request open for
that final, also if the answer starts playing meanwhile; an empty final (rejected noise) joins
nothing and asks nothing again
- The engine checks the cancel before the tool router, the planner and a planned tool call too
- Finals with a context are batched again: each context is cut from its end to the shortest one in
the batch (ctranslate2 needs the start token at one position); ten finals at once 1.3 s
- Measured harsh conditions (docs/v2/stt-accuracy-plan.md): the context never made them worse
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The words of the topic were the ones misheard live (화면공이 for 화면 공유, 계산지 for 계산기), and
five models made the same mistakes. Each final is decoded as before and again with the last 180 s
of the channel's conversation (everyone's finals and Jarvis's speech, at most context_chars=200
characters) as the previous-text prompt; the prompted text is used only when the plain one was kept,
stays within half its letters and keeps every hotword count, as for the hint.
Measured on the game PC (RTX 5070 Ti, 100 recordings of the user, 16 with their own text): CER
12.0% -> 7.5%, empty results unchanged, noise clips with text unchanged (16/108), final decode
p95 170 -> 253 ms. Finals with a context are decoded one clip at a time (ctranslate2 needs the
start token at one position in a batch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live 2026-11-19 every piece of one thought (pauses of 0.1-4.7 s) was answered on its own and the
brain wrote each cancelled answer to its end, so the replies queued up to 35 s.
- conversation: a speaker who goes on before their answer is heard (within call.continueMs,
default 5 s) has the earlier request taken back (policy withdraw, not missed) and the joined
text answered as one request; a stop still stops; no second earcon; partials prepare the joined
text
- voice worker: a reply cancelled while waiting is never started, a running one stops at the
engine's next model or tool call (turn_state.cancel), and it uses up no turn, so a confirmation
carried by the joined request does not lapse
- specs: conversation.spec.md (Going on), voice_worker.spec.md, llm_contexts.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The side menu keeps 대시보드 and 대화 on top and puts the rest in groups that
open when their name is pressed (디스코드, 대화 방식, 음성·모델, 기능, PC, 관리);
the shown screen's group opens and open groups are kept. 채널·호출 is split
into 음성 채널, 호출 and 여럿이 말할 때, and 모델 into 받아 적기·말하기 and 대화
모델; the chat wake words join 호출, the recording switch 음성 채널 and the
default location 기능 켜기·MCP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The default wake words are 비서, 비서야, 비서님, 김비서, 김비서야 and 김비서님.
Where several wake words are heard starting at the same place, the closest
and then the longest is the call, so a call form is removed whole (비서야 멈춰
is a stop) and a near match of a longer word does not eat into what follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The share target checks that the person asked about shares their screen in
Jarvis's voice channel, finds Discord's popped-out window (title not ending in
Discord, the voice channel's name first) and, when there is none, asks for it in
the channel and looks again every second for 30 s. Screen actions now wait as
long as PC actions in the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
restore activates the window (SW_RESTORE), so restoring before keys made the
window already in front: the window that was in front before was not brought
back, and the game / full-screen check came after the window had moved. Keys
now note the window in front and check for a game first, then restore inside
the moment in front. A minimised window never takes keys behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the game PC agent-desktop's press --window-id answers APP_NOT_FOUND for
the new Calculator ("not found with exact process identity", nothing sent),
though focus-window brings it in front. The keys now go to the window in
front while it is still that window. The Windows test checks Notepad's typed
text by element values only (a label such as Ctrl+Shift+X contains "hi").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User decision 2026-11-19 (mixed way). agent-desktop only presses keys in the
window in front, so the app now picks one way before any key is sent:
- behind: one key without modifiers that types or moves is posted to the
window's one shown text control (Edit, RichEdit, RichEditD2DPT, Scintilla);
the person's screen and focus stay. Windows does not tell a background
thread's focused control (win-ci VM: GetFocus is null even attached), so
only windows with exactly one text control get keys behind.
- in front for a moment: combinations, other keys and every other window;
the window that was in front is brought back after (also after a failed
press). A window already in front gets the keys there.
- never in front while game mode is on or the window in front is full screen.
- never both ways, so a key is never pressed twice.
Specs: admin.spec.md "PC control" (Keys), pc-features.md, pc_control.spec.md.
Tests: six new fake-desktop tests in pc.rs; tests/pc_keys_windows.rs on a real
desktop (win-ci VM: letters behind Notepad with Character Map kept in front,
ctrl+a in front and back, the classic Calculator in front and back).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the live conversation on 0.1.40 (2026-11-18):
- The bot browser told pages a program drives it (navigator.webdriver), and
YouTube stopped every video about 45 s in with "문제가 발생했습니다"
(ump.spsrejectfailure, reproduced twice on the game PC). Chrome now starts
with --disable-blink-features=AutomationControlled, as the M0 video check
already did; the live Chrome test checks webdriver is false.
- "네." to "계산기를 실행해도 될까요?" got webSearch and botBrowser from the
router, so the consent call was refused and the calculator never started.
Waiting tools: pcControl stays on offer while a request asked in the
previous reply waits, botBrowser while it holds the asker's page.
- "소리 반으로 줄여 줘" for the bot's video worked on the person's own Edge
window. botBrowser gets a volume action (media elements and YouTube's player),
the page shows its volume, its description names its actions first (the part
the planner sees), pcControl's says the bot's browser is not among its
windows, and a turn whose plan has no step while a tool holds something for
the asker gets the one-step nudge for it.
Evals (qwen3.6 on the game PC): volume 0/3 before, 8/8 after; "네." consent
10/10; pcControl 10/10; botBrowser 16/18 (the NewJeans play case fails 0/6
before this change too). Unit: Python 2542 passed, 13 failed (the same 13 as
before); app 348 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checked against agent-desktop 1.2.0 on the Windows test VM (normal and
elevated sessions); each fault has a test:
- the state folder is made private to the current user: a data folder
outside the user profile inherits Authenticated Users: Modify and
agent-desktop then refused every reading (INTERNAL), which broke all
of PC control on such PCs
- run with the program already open uses the window agent-desktop names
instead of waiting for a new one (failed after 5.3 s, now 0.6 s);
several copies get their own reason
- keys to a window Windows will not bring to the front says so
(NOT_FRONT) and sends nothing
- close that a save dialog holds up, or several copies, get their own
reasons instead of a generic failure
- an action that closes its own window answers done with no elements
- fields a reading showed as secure are refused as password fields
before anything is sent
Verified on the VM: calculator 1 2 + 3 = read back 15, Notepad type,
ctrl+a and a letter, elevated windows refused from a normal session and
normal ones not, the emergency stop killing a running command and
blocking acting while reading works.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
app/src/pc.rs runs every pc_action from the worker (admin.spec.md, "PC
control"): one agent-desktop 1.2.0 run per command with a 15 s limit and
its state in the data folder, never --force or raw input, typed text
only through stdin. Each action is decided by permissions::decide for
its asker (not registered, never, ask without consent), then refused
during the emergency stop, while an anti-cheat game runs (Vanguard,
EAC, BattlEye, FACEIT, GameGuard, XIGNCODE3 and the person's list), on
an administrator window and in a password field. Results carry the
window and its elements read again (200 at most, cut to 120 chars),
agent-desktop's error codes become Korean reasons, and each action is
recorded as pc.<action> with the program's file name only.
The emergency stop is a global hotkey (pcControl.stopHotkey, default
Ctrl+Shift+F12) on its own message loop; it kills the running command
and blocks everything but reading until pressed again or lifted on the
PC control screen. features.pcControl (on by default, nothing happens
until someone is registered) and the pinned exe hash decide whether
the brain gets the tool. The installer ships agent-desktop.exe from the
pinned release archive, both hashes checked.
Tested with a fake agent-desktop (342 lib tests pass) and cargo check
for x86_64-pc-windows-gnu; not yet run against real agent-desktop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brain operates programs through the app (pc_control.spec.md):
windows, read, click, type, keys, run and close, with window text fenced
as untrusted screen data and every action answered from the fresh
reading the app returns.
- Asking first: an action the app wants asked about is not done; the
tool keeps the request and only the person's very next message, judged
by the small model chain (fenced, YES/NO, 600 chars), gives consent,
which then covers those kinds on that window for the rest of the
reply. The model's confirm flag is not what decides.
- Never done: before the first acting call per window and kind in a
reply, a small call checks for payments, secrets, deleting files and
administrator rights; anything but a clear no refuses and nothing
reaches the app.
- The worker keeps each person's pending requests between their
replies, passes the app's ask answers through and waits 45 s for a PC
action (starting a program and reading it takes the app ~20 s).
- The reply engine's repeat guard no longer limits pcControl: pressing
calculator keys is many calls and the same one can be meant twice.
Before, the third call was blocked and the model then claimed to have
pressed a payment button it never pressed.
Eval evals/test_pc_control_tool.py on Qwen3.6 (game PC Strata), 8 runs
of 4 cases after the last changes: calculator 12+3 read back as 15 6/8
(the misses press the digits out of order and say the wrong result read
from the screen), ask then consent in the next message 8/8, a request
written in a window not followed 8/8, payment never pressed and never
claimed 8/8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Asked to talk with no topic ("말해 봐", usually to hear a changed voice,
speed or volume), the model answered with an okay, a promise to speak
("말씀하신 대로 속도를 높여 말씀드리겠습니다") or a question about what to
say. The voice persona now says to pick the topic itself and say it,
never a made-up live fact such as the weather (the first wording made it
reach for the weather).
The said-not-asked eval now also checks that the talk cases talk and
make up no live fact. Measured on Qwen3.6 (game PC Strata, compact prompt
and intent judge), 6 clock variants, base and new interleaved: the five
talk cases failed 23/30 before, 12/30 after. The controls (a real speed
request is not claimed done, a repeat is said again), the greeting and
reply length evals give the same results before and after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The done-by-user guard asked its first yes/no question after every
follow-up answer was made, adding a short LLM call (~0.4 s) before
Jarvis spoke. The question depends only on the user's words, so the
voice worker's prepare now asks it on the partial transcript after the
tool pick (prepare_done_by_user, follow-up turns only) and the guard
uses that judgement (prepared_done), asking only whether the answer
refuses when the user said they did it. Without a prepared judgement,
or when it failed, the guard asks for itself as before. The reply does
not wait for the judgement, only the guard does.
Measured on Qwen3.6 through the game PC's Strata, 12 pairs of short
follow-up questions in alternating order: brain time median 1.06 s
before, 0.69 s after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A native tool call message with text is not an earlier answer: leaving it out
orphaned its role=tool result. Only plain assistant answers are left out.
- The guard is off once any tool run started in the reply, whatever it returned:
an MCP call with an empty result was not in invoked_tools_history, so a second
try could run it again.
- HistoryContext.answers marks the lines that are Jarvis's answers to the asker
(to_id is the asker); only those leave the history, not someone else saying the
same words.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Field turn 2026-11-18: Jarvis said speed control is not supported, the user changed
the speed in the app and said "I adjusted it, so say something", and Jarvis refused
again, often in other words. Prompt wording did not move it (five rules, two history
shapes tried).
v2 replies with an earlier Jarvis answer in the dialogue memory and no tool call now
ask the chat model two short yes/no questions: does the user say they did something
themselves, and only then, does the answer say something cannot be done. On two
yeses the answer is made once more with Jarvis's earlier answers (dialogue memory and
the history lines with the same text) left out of that reply's context.
Real model (qwen3.6 35B on Strata), field case with history and dialogue memory, 20
same-prompt pairs: refused 12/20 before, 1/20 after; "what did you say? say it
again" still said it again 20/20 before and after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tool honesty rules ("if no tool can do it, say so") made the
model refuse actions the user only reported doing themselves: "알람
일곱 시로 맞춰놨어, 잘했지?" got "제가 도와드릴 수 없습니다". The
persona rules and the end-of-prompt reminder now say such a report is
not a request.
Measured on the service model (qwen3.6 35B, app Strata, compact prompt
and intent judge), the same four eval files before and after:
test_said_not_asked, test_schedule_tool, test_greeting_no_tools and
test_screen_view_tool went from 20/24 to 22/24. The alarm case now
passes; the one still failing is the field turn whose history already
holds Jarvis refusing the same thing (it repeats its earlier answer,
four other wordings did not move it). Piranesi fails before and after.
Adds the correctly heard field turn as a case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 12 dB shelf above 1 kHz from 0.1.34 made the user's speech worse.
Re-transcribing the 34 live recordings of 2026-11-18 (17:32 to 19:53)
with the shelf reproduces every live text; without it "왜 내 방해를"
comes back instead of "올해 내 방해를" (the user said "난 왜 라고
했어" right after), "알아서 좀 조정해 놨는데" instead of "아라해석도
좀 조정해놨는데", and "넌 뭐하는 애야?" instead of an empty text. No
utterance got clearly better with it. Reverts 9f62bfb and 04c23f9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Field failure 2026-11-18: "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고"
was answered "말하는 속도 조절 기능은 현재 지원하고 있지 않습니다". The eval
runs the v2 brain setup (compact prompt, intent judge, stop and
chatChannels) on that turn and similar ones, plus a control where a real
request no tool can do must not be claimed done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MeloTTS's Korean front end read "60%" digit by digit ("육영퍼센트") when
the Windows MeCab build did not tag what followed as a bound noun, and its
sentence splitter broke "4.2미터" at the dot ("사. 이미터"). The KR child
now spells decimals with 점, drops thousands commas and gives every other
whole number its Sino-Korean reading, leaving numbers before native
counters (시, 개, 마리, ...) to MeloTTS's own counter rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>