From the live conversation on 0.1.40 (2026-11-18):
- The bot browser told pages a program drives it (navigator.webdriver), and
YouTube stopped every video about 45 s in with "문제가 발생했습니다"
(ump.spsrejectfailure, reproduced twice on the game PC). Chrome now starts
with --disable-blink-features=AutomationControlled, as the M0 video check
already did; the live Chrome test checks webdriver is false.
- "네." to "계산기를 실행해도 될까요?" got webSearch and botBrowser from the
router, so the consent call was refused and the calculator never started.
Waiting tools: pcControl stays on offer while a request asked in the
previous reply waits, botBrowser while it holds the asker's page.
- "소리 반으로 줄여 줘" for the bot's video worked on the person's own Edge
window. botBrowser gets a volume action (media elements and YouTube's player),
the page shows its volume, its description names its actions first (the part
the planner sees), pcControl's says the bot's browser is not among its
windows, and a turn whose plan has no step while a tool holds something for
the asker gets the one-step nudge for it.
Evals (qwen3.6 on the game PC): volume 0/3 before, 8/8 after; "네." consent
10/10; pcControl 10/10; botBrowser 16/18 (the NewJeans play case fails 0/6
before this change too). Unit: Python 2542 passed, 13 failed (the same 13 as
before); app 348 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brain operates programs through the app (pc_control.spec.md):
windows, read, click, type, keys, run and close, with window text fenced
as untrusted screen data and every action answered from the fresh
reading the app returns.
- Asking first: an action the app wants asked about is not done; the
tool keeps the request and only the person's very next message, judged
by the small model chain (fenced, YES/NO, 600 chars), gives consent,
which then covers those kinds on that window for the rest of the
reply. The model's confirm flag is not what decides.
- Never done: before the first acting call per window and kind in a
reply, a small call checks for payments, secrets, deleting files and
administrator rights; anything but a clear no refuses and nothing
reaches the app.
- The worker keeps each person's pending requests between their
replies, passes the app's ask answers through and waits 45 s for a PC
action (starting a program and reading it takes the app ~20 s).
- The reply engine's repeat guard no longer limits pcControl: pressing
calculator keys is many calls and the same one can be meant twice.
Before, the third call was blocked and the model then claimed to have
pressed a payment button it never pressed.
Eval evals/test_pc_control_tool.py on Qwen3.6 (game PC Strata), 8 runs
of 4 cases after the last changes: calculator 12+3 read back as 15 6/8
(the misses press the digits out of order and say the wrong result read
from the screen), ask then consent in the next message 8/8, a request
written in a window not followed 8/8, payment never pressed and never
claimed 8/8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The done-by-user guard asked its first yes/no question after every
follow-up answer was made, adding a short LLM call (~0.4 s) before
Jarvis spoke. The question depends only on the user's words, so the
voice worker's prepare now asks it on the partial transcript after the
tool pick (prepare_done_by_user, follow-up turns only) and the guard
uses that judgement (prepared_done), asking only whether the answer
refuses when the user said they did it. Without a prepared judgement,
or when it failed, the guard asks for itself as before. The reply does
not wait for the judgement, only the guard does.
Measured on Qwen3.6 through the game PC's Strata, 12 pairs of short
follow-up questions in alternating order: brain time median 1.06 s
before, 0.69 s after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A native tool call message with text is not an earlier answer: leaving it out
orphaned its role=tool result. Only plain assistant answers are left out.
- The guard is off once any tool run started in the reply, whatever it returned:
an MCP call with an empty result was not in invoked_tools_history, so a second
try could run it again.
- HistoryContext.answers marks the lines that are Jarvis's answers to the asker
(to_id is the asker); only those leave the history, not someone else saying the
same words.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Field turn 2026-11-18: Jarvis said speed control is not supported, the user changed
the speed in the app and said "I adjusted it, so say something", and Jarvis refused
again, often in other words. Prompt wording did not move it (five rules, two history
shapes tried).
v2 replies with an earlier Jarvis answer in the dialogue memory and no tool call now
ask the chat model two short yes/no questions: does the user say they did something
themselves, and only then, does the answer say something cannot be done. On two
yeses the answer is made once more with Jarvis's earlier answers (dialogue memory and
the history lines with the same text) left out of that reply's context.
Real model (qwen3.6 35B on Strata), field case with history and dialogue memory, 20
same-prompt pairs: refused 12/20 before, 1/20 after; "what did you say? say it
again" still said it again 20/20 before and after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strata reads every reply prompt from the start in 2,048-token steps. The v2
compact prompt carried v1's full tool guidance and constraints (about 900
Qwen3.6 tokens) on every turn with a data tool, and the reply-language
default (about 120 tokens) twice.
COMPACT_TOOL_GUIDANCE keeps the same rules (act instead of refusing, facts
only from the result, look up named things, auto-derived and self-contained
arguments) in about a quarter of the tokens, without English examples; the
reply-language default says the same in half. v1's prompts are unchanged.
Measured on Strata (Qwen3.6, 20 lines of recent conversation): a weather
turn's prompt 3,436 -> 2,722 tokens, a search turn's second call
4,096 -> 3,383. Evals (compact prompt + intent judge, Qwen3.6): 24/24
before, 30/30 twice after (weather location, follow-up arguments, tool-skip
confabulation, search language, reply language, Possessor fidelity, reply
length, multi-turn context).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the one nudge, a second answer without the tool no longer goes out
as it is: the turn takes the planned path (the planner runs then, one
extra call only on this failure) and the skipped-tool guard runs the
planned tool, so a made-up "24 degrees" answer cannot reach the user. A
plan that needs no tool (a question back to the user) still lets the
answer go out.
Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs):
the five tool-skip/weather/screen/follow-up/router files 72/84 (71/84 on
0.1.20, same known failures); Possessor director-then-filmography 0/3,
also 0/3 on 0.1.20 (qwen3:8b stops after one search; webSearch turns are
still planned).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the router's only pick is a tool that does its job in one call
(getWeather, logMeal, fetchMeals, screenView) and the chat model calls
tools natively, the planner no longer runs: on Strata it averaged 2.5 s
per tool turn (2026-11-16 16:45-16:58 engine log, 9 calls) before the
answer could start. Search, browsers, schedule, notes, chat channels and
picks of two or more tools still plan. The rule reads the router's pick,
not the words, so it is language-agnostic.
Without a plan the skipped-tool guard has nothing to run, so these turns
get one nudge instead: an answer given before any tool ran is dropped and
the model is asked to call the tool (or ask for a missing detail).
Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs of
tool_skip_confabulation, weather_autoderive_location, screen_view_tool,
followup_supplies_missing_tool_arg, tool_router_implicit): 71/84 before,
72/84 after, same known failures; planner calls 14 -> 3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- botBrowser `play`: searches YouTube for the words as said, opens the first video (not an ad,
without the playlist) and starts it, in one call (an address given opens that page); `type`
also takes `query`. Live 0.1.19: the model stopped on YouTube's home page or the results, a
retry was refused as a third call, and the answer still said the video played.
- `brain.location` (기본 위치, default 대한민국) goes to the brain as `location_name`: the time line
says it without GeoIP (no database on the game PC), and getWeather without a place uses it.
- Planner and webSearch keep search words in the user's language (never translated into
English), so recommendations fit where the user is.
Evals (qwen3:8b, 3 runs): play ending on a playing video found with the words as said 0/6 -> 5/6;
Korean searches for Korean requests 6/9 -> 8/9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- screenView tool (screen_view.spec.md): whole screen, monitor by number, window by part of its
title or program, or the list of monitors and windows; fast, normal or detailed pictures.
- The app captures with Windows Graphics Capture (app/src/screen.rs, admin.spec.md "Screen
viewing"), decides for the asker (only people registered for PC control), never picks, lists
or shows excluded programs (pcControl.screenExclude, password managers by default), refuses
minimised windows and black pictures, and records screen.view in the activity log.
- Pictures ride on the tool message (ToolExecutionResult.images); llm.py sends them to
OpenAI-compatible servers as image parts after the tool messages. They stay in that reply only:
not in the tool carryover, logs or prompt dumps.
- The LLM reads pictures only when screen viewing is available (--vision gpu normally, cpu in a
game), measured apart for the placement; otherwise screenView is a disabled tool.
- Worker: screen_action / screen_action_result, 4 MiB messages.
- Eval evals/test_screen_view_tool.py; Windows check tests/screen_windows.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A personalNotes tool (add, list, delete the asker's own notes) offered only
with a notes backend. The app keeps the notes in config/notes.json and sends
the asker's with every voice, chat and task reply; the reply engine puts them
in a <person_notes> fence as that person's preferences, which never permit an
action or a tool. The worker's schedule bridge becomes a shared app bridge
for schedule_action and note_action. A 사람별 지침 screen edits and deletes
any note; features.notes turns the tool and the notes off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brain gets a schedule tool (add, list, delete the asker's own reminders,
timers and tasks) offered only with a schedule backend. The voice worker asks
the app through schedule_action and runs fired tasks as reply via task with
read-only tools, no MCP and a fresh short-term memory. The app keeps the
schedules in config/schedules.json, fires them every 5 s, speaks them when the
person is in Jarvis's voice channel and otherwise posts in the chat channel
they asked in (or schedule.chatChannel for voice). A feature switch and a
예약 screen list and delete everyone's schedules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Discovery runs outside the MCP tool cache's lock (one at a time), so a turn reading the cached
tools never waits for a slow server; a new conversation refreshes in the background.
- The log hub masks secrets before a line is kept or written, so a key replaced or removed later
stays hidden on the screen, in copies, files and exports, also after a restart.
- The worker takes JARVIS_MCP_ENV out of its environment at start, before MeloTTS runs, and hands
it to the brain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Settings `features` (weather, webSearch, meals) and `mcpServers` (name, command, args, env names,
enabled), edited on the new 기능 screen. The tools of a feature that is off go into the brain's
`disabled_tools`; only enabled MCP servers go into its `mcps`.
- MCP environment values (API keys) live only in the encrypted secrets (`mcp:<server>:<NAME>`),
shown as stored or not, masked in logs, removed with their server, and reach the worker only in
`JARVIS_MCP_ENV`, which the brain takes out of its environment before giving each server its own.
- `enabled_builtin_tools(cfg)` is the one list the router, toolSearchTool and the executor use, so
a turned-off tool is neither offered nor run, also when the model names it.
- The v2 brain never offers v1's `localFiles` (home folder read/write/delete) and `screenshot`:
anyone in the voice or chat channels could ask for them. Screen viewing comes in M7.
- The brain connects MCP servers on its own thread and logs each outcome without debug logging.
- Stored secrets are masked in the log hub as soon as the data folder is found.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every <parameter> value was passed as text, so a delete confirmed with
confirm=true reached chatChannels as "true" and asked again. Values now take
the type the tool's input schema declares (boolean, integer, number, array,
object); a value that does not fit stays text. The roadmap's eval summary now
names the reply-length failure apart from the later passing runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A chatChannels pick from the router counts as a tool turn for the intent
judge ("post this in the schedule channel" needs the planner).
- The default-language directive names an explicit language request as its
one exception; the history block says plans and appointments people
mentioned need no calendar tool; why/how answers give only the core reason.
Evals (Qwen3.6, v2 options): reply length 3/3 (v2 185-205 chars, v1 315-356),
language, history, chat channels and tool-skip 22/22; full suite 219 passed,
6 failed (paris/london, graph branch and four v1 single-user memory cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- compact_prompt: the v2 persona in a tenth of v1's words, answers as short as
possible, the full tool guidance only on turns with a data tool. A greeting's
reply prompt went from about 4,500 to 1,400 tokens on Strata.
- intent_judge: the tool router judges whether the turn needs tools; the
planner runs only when it does. prepare_intent runs the pick ahead on a
partial transcript (worker message prepare), and the reply reuses it.
- Without stream_browser the screen-share tools are not offered.
- The time context carries the 12-hour form (10:37 was read as pm).
- A <tool_call> block written as the reply is parsed as a call, not spoken.
Evals (Qwen3.6 on Strata, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1):
221 passed, 4 failed (known failures); new evals/test_voice_reply_length.py
(v2 answers 234 chars against v1's 416) and test_reply_language_default.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brain passes default_language (worker --reply-language, default Korean) to
the reply engine; unlike the output_language lock it gives way when the user
explicitly asks for a language. Tools get the ko code (--tool-language).
Eval evals/test_reply_language_default.py (Qwen3.6 on Strata): 6/6, English
questions answered in Korean, an explicit English request in English, and the
control without the default answers English in English.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Drop the engine's deterministic Korean weather shortcut. It bypassed the
chat model and the tool runner, so a location the user had stated could
never be used. The tool already returns one clean Korean sentence.
- getWeather and the tool-argument prompts pass the place the user has
said they are (conversation or memory) and leave the argument empty
otherwise, so the detected location is not copied into it.
- getWeather retries a "district, city, country" name with its leading
parts dropped when the geocoder finds nothing.
- The planner gets the live time/location line and plans
searchMemory topic='where the user lives' only when the location is
unknown or disabled.
- The per-reply repeat guard counts only this reply's tool messages, so
calls carried over from the previous reply no longer block a retry the
user asks for.
- Memory extractor: the context block sits after the examples and the
no-location example uses a different query, so a location already in
context is not asked for again. New counterpart eval checks that it is
still asked for when the context has none.
- Tool retry eval fails every call in turn 1, so a model that retries by
itself inside turn 1 still exercises the user's "try again".
- Specs (planner, reply), docs/llm_contexts.md and the roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Tool (chat_channels.spec.md): list, read, search, send, edit, react,
delete, pin, unpin. Offered only when the v2 brain gives the turn a chat
backend, and then on every turn. Channel by ID, name, part of a name or a
name inside the text; a message ID finds its channel; planner forms of the
action ("send_message") work. Read and search results are fenced as
untrusted data. delete only reports until called again with confirm=true
in a later turn of the same person.
- Worker: chat_channels from the app, ChatBridge reads/searches the history
store and sends chat_action, waiting for chat_action_result (15 s timeout,
failed on session close); per-person reply turns and pending deletes.
- App: channel switches edit/react (on) and delete/pin (off); chat_channels
sent on worker ready and after login; chat_action runs send, edit (own
messages only), react, delete, pin, unpin and answers with ids or the error.
- Engine: chatChannels left out of the catalogue without a backend; the
"router returned everything" check now tolerates that.
- live_discord: edit own, refuse editing the speaker's, react, pin (Discord
refuses in a voice channel's chat, reported as an error), delete, refuse an
unregistered channel; passed in 109.4 s.
- Eval (qwen2.5:7b, 3 runs): 10 of 12; never obeyed the "delete everything"
message and never deleted in the same turn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A final is stored under the channel, speaker name and time fixed when its
end was decided. Decoding takes a while; a move, rename or late joiner
meanwhile used to change where and when it was said, and who heard it.
- At start the worker closes presence a killed run left open
(HistoryStore.close_open_presence), so a person who is gone no longer
hears new speech after a restart.
- run_reply_engine(legacy_memory=False) skips v1's single-user memory (warm
profile, extractor, diary, graph, digest); the v2 brain uses it so another
user's profile or diary never reaches the prompt.
- While a brain reply runs, stdout and stderr are muted (also helper threads
such as the web fetch pool); the worker logs through voice_worker/log.py to
the original stderr.
- Live check: Ollama's /v1 fixes the context at 2,048 tokens and cut the
prompt in earlier runs; with an 8k tag the chat-record question was answered
4 of 4 and nothing reached the output. Noted in llm_contexts and the eval.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Search scored only the newest 5,000 matching records, so an old record
behind enough newer chatter was never found. Both passes (word rarity counts
and the score) now run in SQLite over every record in scope and only the
best rows come back.
- context() excluded the recent window from search and then cut that window
for the budget, so a relevant recent record could vanish. Search now covers
the recent window, matches keep their half of the budget, and a matched
record is removed from the recent block so it appears once.
- Matching used SQLite's ASCII-only lower() against Python's lowercasing.
Stored text gets a folded column (NFKC + casefold) and query words are
folded the same way, so case is ignored in every script.
- Prompt label "Records related to this question" (matches may now be recent);
eval re-run 3x on qwen2.5:7b, all pass. Spec, memory.md and roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/jarvis/memory/history.py keeps voice transcripts, Jarvis's answers,
registered chat channel messages (with edits and deletes) and voice channel
presence in one SQLite file. Every read is scoped to the asker: own records
always, chat channels and others' words (only while the asker was in the
same voice channel) behind settings, plus an optional since and off.
- Search is language-agnostic: Unicode words, prefixes keeping 60% of the
word, rarer words weigh more, newer wins ties. 100k records search in
24-61 ms here. context() returns the recent window and the matches cut to a
token budget, matches at most half so chatter cannot push out the record
the question is about.
- run_reply_engine(history=...) puts that context into the system prompt
under "Information shared in previous conversations", fenced as data.
evals/test_history_context_recall.py: with the block qwen2.5:7b answers the
stored day in Korean and English (3 of 3 runs), without it the day is unknown.
- Spec history.spec.md, registry entry in CLAUDE.md, reply.spec.md,
docs/llm_contexts.md, docs/v2/memory.md and roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the skipped-tool guard could not run the planned call (not a literal
call, or arguments the tool does not take) it delivered the answer it had
held back, which was given without the tool and could be a made-up value.
The planned call is now recorded as a failed tool result and the model
answers from that, so the user hears that it could not check or do it.
M0 kit: a case passes only when the final answer uses the tool result
(13 degrees, not a made-up 24) or, for the no-tool request, says it
cannot do it; the no-tool request now counts. The kit mirrors the
engine's failed-result path, and a failing model moves on to the next
candidate on these stricter checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M0 on Strata caught both v2 models skipping a tool they had been given:
the recommended model made up "24 degrees, sunny" instead of calling the
weather tool, and the fast model said "timer set" without calling the
timer tool (3 of 3 each). qwen2.5:7b does the same through the engine.
- Prompt: TOOL_HONESTY_RULES in the persona (live facts only from a tool
result this turn, actions only claimed after a tool reports success, no
answer text next to a tool call), restated at the end of the system
message by tool_honesty_reminder() with the turn's real tool names.
- Engine: skipped-tool guard. When a native-tool model answers before any
tool has run and the plan names a callable tool as a concrete call, the
answer is dropped and the plan steps are direct-executed, then the model
answers from the results. Only literal, schema-valid steps run (regex
fast path, no LLM repair); otherwise the held-back answer is delivered.
- Planner: the AVAILABLE TOOLS lines now carry argument names
(planner_tool_catalog), so concrete steps use the tool's real keys.
- Eval evals/test_tool_skip_confabulation.py; on qwen2.5:7b, 3 runs each,
available-tool cases went from 0/12 to 9/12 calling the tool.
- M0 kit: tools_fast / tools_recommended stages (run_m0_tools.bat) rerun
tool calling before and after the fix on the real models, loading the
production system_prompt.py and planner.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only
exposes the worst case where the chat model rambles or loops. Add an
ollama_num_predict config (default 512, 0 disables) wired into the reply
loop's chat call on both the native- and text-tool paths. The 512-token
headroom stays well above this app's short tool-call JSON, so capping never
truncates a tool call. This keeps the user's quality model instead of
downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop Markdown files into an agents/ folder and their contents are appended to
the main reply LLM's system prompt, so an operator can extend the assistant's
rules/tone without code changes. Files are concatenated in filename order
(use 00-, 10- prefixes to control ordering) and re-read once per turn, so edits
apply on the next reply with no rebuild/restart. Fail-open: a missing, empty,
or unreadable folder yields no instructions and never breaks a reply.
- load_agent_instructions() in system_prompt.py (AGENTS_DIR env, default
/app/agents); reads *.md only, skips blanks, ignores non-dir paths
- engine.py appends it alongside the existing settings-UI llm_instructions,
under the same "Additional instructions from the operator:" framing
- docker-compose.yml bind-mounts ./agents:/app/agents:ro and sets AGENTS_DIR
- agents/example.md.sample starter template (.sample is not loaded)
- tests cover ordering, md-only filtering, blank-skip, env/arg resolution,
and fail-open paths
- README, .env.example, docs/llm_contexts.md updated
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The persona prompt was built from the raw OUTPUT_LANGUAGE env while the
reply-language directive read the settings-web UI value (config JSON), so
changing the language in the settings page was honoured by the directive
but ignored by the persona, leaving them contradicting each other.
Add _resolve_output_language() as the single source of truth (config wins
over env) and feed the same resolved value to both build_system_prompt()
and reply_language_directive(). Update docs/llm_contexts.md to match.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The settings-UI output_language was ignored because the OUTPUT_LANGUAGE env took
precedence. Read the config value first, fall back to env, so changing the
language in /settings actually applies.
Adds /settings (served by the bridge) to change the LLM model (from installed
Ollama models), Whisper model, TTS engine + MeloTTS speed, output language,
agentic max-turns, thinking mode, and free-form LLM instructions — live, with a
'apply' that restarts the bridge + TTS worker. Settings persist to the runtime
config JSON; engine reads output_language + llm_instructions and the TTS worker
reads melo_speed from it. Bridge port publishable for access.
GeoIP auto-detect is unavailable in the container, so '부산 날씨' failed
(no location). Extract the city from the utterance and pass it to getWeather.
Also report navigate by site name instead of the mid-load about:blank url.
The 7B narrated '구글 메인으로 돌아갑니다' without acting, so the screen stayed on
Naver. Split site intent into SEARCH vs NAV: nav words (돌아가/이동/열어/메인/go
back) now drive controlBrowser.navigate to the site homepage directly, search
words run controlBrowser.search — both deterministically, no LLM.
getWeather now returns only the Korean sentence (지금 <곳> 날씨는 <상태>, 기온 N도
(체감 M도)입니다) with no English/°C source. A deterministic weather path in the
engine returns it verbatim, bypassing the 7B which was rephrasing into multiple
sentences and leaking 'Celsius'.
qwen2.5:7b leaked Chinese/Cyrillic mid-reply despite the OUTPUT_LANGUAGE
lock, which was buried mid-prompt. Repeat the lock at the END of the system
prompt (recency) and ban specific foreign scripts explicitly.
- Site-specified search ("네이버에서 X 검색해줘") now runs controlBrowser.search
directly in the engine when broadcasting, instead of relying on the 3B model
to emit the tool call (it kept narrating "검색하겠습니다" without acting).
- Set OUTPUT_LANGUAGE=ko so replies are Korean-only — stops the small model
leaking CJK/Hanja and English fragments (每, 朗, "feels like") into weather
and other answers, and keeps them concise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Greetings/small-talk routed no data tool yet still ran the episodic memory
enrichment (LLM keyword extract + diary/graph search, ~1s) every turn. Skip it
when the router picked no external-data tool — the always-injected warm profile
still personalises the reply. Also drop the voice silence-detection wait
800ms -> 600ms for snappier turn-taking. Warm "안녕" now lands well under the
3-4s target.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of "weather/search do nothing": the engine forced TEXT tool-calling
for all <=7B models, but qwen2.5:3b emits clean NATIVE tool calls and fails at
the text format — so it just narrated ("부산 날씨는 맑습니다") and never called
getWeather/webSearch/controlBrowser. Use native tool-calling for tool-capable
small families (qwen2.5/qwen3/llama3.x/mistral); native still auto-falls back
to text on HTTP 400, so non-tool models (gemma) are unaffected.
Also launch Chrome with --test-type (removes the "--no-sandbox unsupported
flag" infobar) and disable the Translate feature/popup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 3B model kept choosing webSearch over controlBrowser even when offered, so
'네이버에서 X 검색' still used the invisible web path. When broadcasting and the
user explicitly names a site, remove webSearch from the allow-list so the
on-screen browser is the only search route.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The small router reflexively routed every "search/open" intent to webSearch
and never surfaced controlBrowser, so "네이버에서 X 검색해줘" did nothing on the
broadcast. Union controlBrowser (+browseAndPlay) into the allow-list every
turn in screen-share mode (like setBroadcast), and steer the model in the
system prompt to prefer the on-screen browser over webSearch when available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
qwen2.5:3b emits tool calls in text shapes the parser dropped, breaking
two reviewer-reported behaviours:
- `getWeather: {"location": "Seoul"}` (a JSON object after the colon) was
dumped wholesale into {"query": "{...}"}, so `location` never reached the
tool. getWeather then ran with empty args, returned the auto-detected
location's weather, the model noticed the mismatch and retried — looping up
to 8 times before giving up with an English error. Now the JSON object after
the colon is parsed directly as the argument dict.
- `call_stop: {"id":..., "function": {"name": "setBroadcast",
"arguments": "{\"action\": \"stop\"}"}}` — a single tool_call object without
the `tool_calls: [...]` array wrapper, behind a `call_xxx:` label — matched
no form, so the raw JSON leaked to the user AND setBroadcast never ran
("방송 꺼줘" did nothing). Now name + arguments are pulled from the embedded
`function` object when the name is in the allow-list.
Field-captured from the live qwen2.5:3b brain (2026-06-12). Tests cover both
shapes, non-ASCII args, dict/string arguments, and unknown-tool rejection.
Harden the reply-language lock so qwen2.5:3b reliably stays in the locked
language instead of leaking the query language back in:
- reply_language_directive(): single resolver with clear precedence —
explicit OUTPUT_LANGUAGE lock wins over the Piper/Chatterbox English-only
fallback (this deployment's actual TTS is Korean MeloTTS, so the legacy
English lock was both wrong and contradicting the Korean lock).
- Stronger, override-explicit directive wording, inserted near the FRONT of
the system prompt so a small model gives it primacy over the persona.
- build_system_prompt(output_language=...): rewrite the persona's "in the
user's language" clause to the locked language so the persona stops
fighting the lock.
- docs/llm_contexts.md: document the resolver, precedence, and placement.
Live-verified on the running brain (qwen2.5:3b): Korean voice-style input
and a cold English query both return fully Korean replies with no CJK/Hanja
leak. Tests cover unset/set/agnostic/whitespace + precedence + persona rewrite.
Add an optional OUTPUT_LANGUAGE env var that forces every reply into a
single language. When set, output_language_directive() injects a "respond
only in <language>" instruction (also forbidding other scripts) into the
chat loop's system prompt, next to the existing TTS English-only lock.
Empty (default) keeps the multilingual "reply in the user's language"
behaviour, so upstream is unaffected.
For the Korean-only deployment this also suppresses the occasional trailing
CJK/Hanja fragment qwen2.5:3b leaks on free-form chit-chat.
- system_prompt.py: language-agnostic output_language_directive() helper
- engine.py: read OUTPUT_LANGUAGE, append directive in _build_initial_system_message
- docker-compose.yml + .env.example: document/pass the new var
- docs/llm_contexts.md: note the new gating on the main reply context
- tests: cover unset/set/agnostic/whitespace cases
Addresses review findings on the dockerized stack:
- Container Chrome search was dead: add --remote-debugging-port + a non-default
--user-data-dir (Chrome 136+ refuses CDP on the default profile), add the
playwright dep (browse-search.mjs connectOverCDP) with browser download
skipped, and connect to 127.0.0.1 not "localhost" (container localhost -> ::1
while Chrome binds IPv4). Verified: browse-search returns real results.
- Broadcast toggle reliability: always offer setBroadcast in screen-share mode
(the embedding/keyword router dropped it for non-English utterances) and make
its description force a tool call. "방송 꺼줘"->stop now 5/5; no false triggers.
- Stop the broadcast on voice leave (no orphaned stream).
- Security: bind VNC/noVNC to loopback by default (VNC_BIND override) and the
bridge to the container loopback (BRIDGE_HOST=127.0.0.1), not published.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>