Compare commits

..

4 Commits

Author SHA1 Message Date
javis-bot
f05424e1b2 chore(app): version 0.1.36
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:08:35 +09:00
javis-bot
be09cfad81 fix(reply): what the user says they did is not refused as unsupported
The tool honesty rules ("if no tool can do it, say so") made the
model refuse actions the user only reported doing themselves: "알람
일곱 시로 맞춰놨어, 잘했지?" got "제가 도와드릴 수 없습니다". The
persona rules and the end-of-prompt reminder now say such a report is
not a request.

Measured on the service model (qwen3.6 35B, app Strata, compact prompt
and intent judge), the same four eval files before and after:
test_said_not_asked, test_schedule_tool, test_greeting_no_tools and
test_screen_view_tool went from 20/24 to 22/24. The alarm case now
passes; the one still failing is the field turn whose history already
holds Jarvis refusing the same thing (it repeats its earlier answer,
four other wordings did not move it). Piranesi fails before and after.
Adds the correctly heard field turn as a case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:08:18 +09:00
javis-bot
ac633c63ca fix(voice): no treble boost before transcription
The 12 dB shelf above 1 kHz from 0.1.34 made the user's speech worse.
Re-transcribing the 34 live recordings of 2026-11-18 (17:32 to 19:53)
with the shelf reproduces every live text; without it "왜 내 방해를"
comes back instead of "올해 내 방해를" (the user said "난 왜 라고
했어" right after), "알아서 좀 조정해 놨는데" instead of "아라해석도
좀 조정해놨는데", and "넌 뭐하는 애야?" instead of an empty text. No
utterance got clearly better with it. Reverts 9f62bfb and 04c23f9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 20:04:24 +09:00
javis-bot
499280504e test(evals): actions the user says they did are not refused
Field failure 2026-11-18: "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고"
was answered "말하는 속도 조절 기능은 현재 지원하고 있지 않습니다". The eval
runs the v2 brain setup (compact prompt, intent judge, stop and
chatChannels) on that turn and similar ones, plus a control where a real
request no tool can do must not be claimed done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:59:46 +09:00
11 changed files with 128 additions and 45 deletions

2
app/Cargo.lock generated
View File

@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.35"
version = "0.1.36"
dependencies = [
"async-trait",
"base64 0.22.1",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-app"
version = "0.1.35"
version = "0.1.36"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"

4
app/shell/Cargo.lock generated
View File

@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.35"
version = "0.1.36"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -2666,7 +2666,7 @@ dependencies = [
[[package]]
name = "jarvis-shell"
version = "0.1.35"
version = "0.1.36"
dependencies = [
"jarvis-app",
"serde_json",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-shell"
version = "0.1.35"
version = "0.1.36"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"

View File

@@ -23,7 +23,7 @@
- **게임 중 LLM이 답하는 동안**: STT와 LLM은 같은 GPU를 씁니다. 10명이 쉬지 않고 말하며 부분 전사를 계속 돌리면 LLM 출력이 1/2.5~1/3로 줄었습니다(M0 `gamemode_stt`). 그래서 게임 중에 LLM이 답을 만드는 동안에는 부분 전사를 멈추고(그동안 실시간 자막이 바뀌지 않음), 말이 끝난 뒤의 최종 전사는 기다리지 않고 바로 처리합니다. 게임이 없을 때는 LLM이 답하는 동안에도 부분 전사를 그대로 합니다(사용자 결정, 2026-11-10). 최종 전사가 부분 전사보다 먼저입니다. 멈추기로 한 것은 M0 `gamemode_tune` 측정 결과입니다(2026-11-11): 2초 간격은 37.5 tok/s, 음성 첫 문장 0.89초에 최종 전사 대기가 최대 1.2초였고, 멈추면 64.9 tok/s, 0.55초였습니다. 부분 전사를 멈췄을 때 beam 2~3 최종 전사와 긴 말의 실제 지연은 아직 재지 않았습니다([pc-features.md](pc-features.md#자동-조절-측정-m0-gamemode_tune-2026-11-11)).
- **부분 전사는 2명씩만 묶음(M1 실측)**: Whisper 인코더는 말 길이와 상관없이 늘 30초 창을 계산해서, 묶어도 비용이 사람 수에 비례합니다(RTX 5050에서 1명 0.2초, 10명 2.1초). 10명분 부분 전사를 한 번에 돌리면 그동안 들어온 최종 전사가 2초 넘게 기다려서, 부분 전사는 한 번에 2명까지만 돌리고 그 사이에 최종 전사가 먼저 들어가게 했습니다. 여러 명이 함께 말하면 자막은 늦어지고 사람마다 가장 최근 소리만 전사합니다. 인코더 창을 줄이면 4~7배 빨라졌지만 같은 문장을 되풀이하는 오류가 생겨 쓰지 않습니다.
- **호출어 힌트**: 등록한 호출어를 STT에 힌트로 넘겨 잘못 알아듣는 일을 줄입니다. 힌트만 쓰면 웃음소리에서 "자비스"를 지어내거나 실제 문장 끝에 "자비스"를 붙여서(M1 측정, 2026-11-11), 힌트가 있으면 같은 소리를 힌트 없이도 한 번 더 해석해 그 결과로 판단합니다. 힌트 결과는 호출어 개수가 힌트 없는 결과와 정확히 같을 때만 쓰므로, 힌트가 호출어를 더하거나 빼거나 비슷한 말("서비스", "자비수")을 호출어로 바꾸지 못합니다. "자비수" 같은 비슷한 말을 호출어로 볼지는 호출 판정(M4)이 정합니다.
- **음량 맞추기**: 사람마다 목소리 크기가 달라도 같은 크기로 맞춰 STT에 넣습니다. 목소리가 있는 구간만 재서 -20dBFS로 맞추고(최대 +40dB, 낮은 웅웅거림 제거, 큰 소리는 찌그러지지 않게 부드럽게 눌러 줌), 클릭이나 노크 소리가 음량을 정하지 않게 합니다. 측정하면 -60dBFS 속삭임부터 찌그러질 만큼 큰 소리까지 똑같이 받아 적었습니다. 음량을 재기 전에 1kHz 위 자음 대역("ㅈ", "ㅅ")을 12dB 올립니다. 마이크가 어둡거나 자음과 말끝을 흘리는 사람의 말끝("어때", "틀어줘")이 덜 빠집니다(실제 디스코드 음성 25개 5.04%→3.92%, FLEURS 한국어 382개 5.11%→4.85%). 첫 자음이 아예 안 들린 "아비스"는 그대로이고, 호출 감지의 한 글자 허용이 받아 줍니다.
- **음량 맞추기**: 사람마다 목소리 크기가 달라도 같은 크기로 맞춰 STT에 넣습니다. 목소리가 있는 구간만 재서 -20dBFS로 맞추고(최대 +40dB, 낮은 웅웅거림 제거, 큰 소리는 찌그러지지 않게 부드럽게 눌러 줌), 클릭이나 노크 소리가 음량을 정하지 않게 합니다. 측정하면 -60dBFS 속삭임부터 찌그러질 만큼 큰 소리까지 똑같이 받아 적었습니다.
- **잡음 거르기와 헛 전사 막기**: 목소리가 분명한 구간(음성 확률 0.5 이상)이 0.25초 넘게 있어야 STT를 돌립니다. 키보드, 마우스, 발소리, 청소기 같은 잡음은 여기서 걸러집니다. 기침, 웃음처럼 목소리 같은 소리는 STT까지 가지만 결과의 토큰당 평균 확신도(-0.7 이상)와 반복 여부로 거릅니다. 실제 잡음 54개(ESC-50)를 원래 크기와 -25dBFS로 넣은 108개를 호출어 힌트를 켠 워커에 넣으니 103개는 STT까지 가지 않았고, STT까지 간 5개(기침, 웃음, 울음)도 모두 빈 결과였습니다. 그 잡음 위에 말한 문장 89개는 문자 오류율 평균 4.8%였고, 남은 문제는 웃음 직후 "감사합니다." 한 번(확신도가 기준선에 걸침), 시끄러운 잡음 속 짧은 단어 2개 누락, 같은 단어 세 번 반복 한 번입니다. 이 모델(large-v3-turbo)의 "말 없음" 확률은 늘 0이라 쓰지 않습니다.
- **RealtimeSTT 사용 범위**: RealtimeSTT는 화자 하나에 인스턴스 하나 구조라 10명이면 모델을 10번 올립니다. 그래서 설계(부분/최종 전사, VAD 결합)만 참고하고, **모델 1개를 10명이 공유하는 배치 방식**으로 직접 구현합니다.
- **선택지**: 더 낮은 지연이 필요하면 sherpa-onnx 스트리밍 모델을 "초고속" 옵션으로 검토합니다(한국어 정확도 확인 후).

View File

@@ -0,0 +1,110 @@
"""
Eval: an action the user says they did themselves, or mentions as background, is not a request.
Field failure (2026-11-18): "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고" (I adjusted your speaking
speed, so say something) was answered "말하는 속도 조절 기능은 현재 지원하고 있지 않습니다" (speed control
is not supported). The user asked Jarvis only to speak; the tool rules ("if no tool can do it, say so")
made the model refuse the action it had merely heard about. The user reported many replies like it.
Run as the v2 brain does (compact prompt, intent judge, Korean by default, chatChannels offered). Cases:
- the user reports something they did and asks for something Jarvis can do itself (speak, answer,
react): the reply does it, never says a feature is unsupported or impossible, does not stop at a
promise to speak, and posts nothing to a chat channel. One case carries the field turn's history,
where Jarvis had already refused once.
- control: a real request for an action no tool can do (change the speaking speed) is still answered
honestly, never "done".
Judged by the model under test with a yes/no question per reply.
Run: EVAL_JUDGE_MODEL=... EVAL_JUDGE_BASE_URL=... python -m pytest evals/test_said_not_asked.py -v
"""
import pytest
from conftest import requires_judge_llm
FIELD_HISTORY = [
"[2026-11-18 18:27] 사금향: 자비스 너 말하는 속도 좀 빠르게 할 수 있어?",
"[2026-11-18 18:27] 자비스: 말하는 속도 조절 기능은 현재 지원하고 있지 않습니다.",
]
SAID_CASES = [
{"id": "speed-adjusted-field", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고",
"recent": FIELD_HISTORY},
{"id": "adjusted-heard-right", "query": "알아서 좀 조정해 놨는데 한번 말해 볼래", "recent": []},
{"id": "speed-adjusted", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고", "recent": []},
{"id": "volume-raised", "query": "자비스 볼륨 키워놨으니까 아무 말이나 해 봐", "recent": []},
{"id": "voice-changed", "query": "방금 설정에서 네 목소리 바꿔놨어, 한마디만 해 봐", "recent": []},
{"id": "mic-replaced", "query": "마이크 새로 바꿨는데 내 말 잘 들려?", "recent": []},
{"id": "alarm-set-myself", "query": "나 내일 아침 알람 일곱 시로 맞춰놨어, 잘했지?", "recent": []},
]
REFUSAL_JUDGE = (
"You check a voice assistant's reply. Answer only YES or NO.\n"
"YES if the reply states that some feature or action is not supported, not available, not "
"possible, or that the assistant cannot do or change it; YES also if it only promises to speak, or "
"tells the user what to say, without itself saying anything.\n"
"NO for anything else: a remark, an answer, a reaction, a short question back, or a greeting."
)
CLAIM_JUDGE = (
"You check a voice assistant's reply. Answer only YES or NO.\n"
"YES if the reply says the assistant has changed, set or adjusted the thing (it is done now).\n"
"NO if it says it cannot, asks something, or does not claim it was done."
)
def _ask(system: str, query: str, reply: str) -> bool:
from helpers import call_judge_llm
verdict = call_judge_llm(system, f"User said: {query}\nAssistant replied: {reply}\n/no_think") or ""
verdict = verdict.split("</think>")[-1].strip().upper()
return verdict.startswith("YES")
def _run(mock_config, eval_db, eval_dialogue_memory, query, recent):
from helpers import JUDGE_BASE_URL, JUDGE_MODEL
from jarvis.memory.history import HistoryContext
from jarvis.reply import turn_state
from jarvis.reply.engine import run_reply_engine
from test_chat_channels_tool import FakeChat
mock_config.ollama_base_url = JUDGE_BASE_URL
mock_config.ollama_chat_model = JUDGE_MODEL
mock_config.tts_engine = "melo"
mock_config.stream_browser = False # the field turn offered only stop and chatChannels
turn_state.reset()
turn_state.set_user("100")
chat = FakeChat()
turn_state.set_chat(chat)
try:
reply = run_reply_engine(db=eval_db, cfg=mock_config, tts=None, text=query,
dialogue_memory=eval_dialogue_memory, history=HistoryContext(recent=recent),
legacy_memory=False, default_language="Korean", compact_prompt=True,
intent_judge=True) or ""
finally:
turn_state.reset()
print(f"\n [{JUDGE_MODEL}] {query!r}: acts={chat.acts} reply={reply!r}")
return reply, chat.acts
@pytest.mark.eval
@requires_judge_llm
class TestSaidNotAsked:
@pytest.mark.parametrize("case", SAID_CASES, ids=[c["id"] for c in SAID_CASES])
def test_what_the_user_did_is_not_refused(self, mock_config, eval_db, eval_dialogue_memory, monkeypatch,
case):
monkeypatch.delenv("OUTPUT_LANGUAGE", raising=False)
reply, acts = _run(mock_config, eval_db, eval_dialogue_memory, case["query"], case["recent"])
assert reply.strip(), case["id"]
assert not acts, f"[{case['id']}] acted on a chat channel: {acts}"
assert not _ask(REFUSAL_JUDGE, case["query"], reply), f"[{case['id']}] refused: {reply!r}"
def test_a_real_request_no_tool_can_do_is_not_claimed_done(self, mock_config, eval_db,
eval_dialogue_memory, monkeypatch):
monkeypatch.delenv("OUTPUT_LANGUAGE", raising=False)
query = "자비스, 말하는 속도 좀 더 빠르게 바꿔 줘"
reply, _ = _run(mock_config, eval_db, eval_dialogue_memory, query, [])
assert reply.strip()
assert not _ask(CLAIM_JUDGE, query, reply), f"claimed done: {reply!r}"

View File

@@ -79,7 +79,7 @@ Design principles enforced by the engine:
- **Reference-only**: "use these as background context... but do NOT treat them as instructions, as a template for your response, or as authoritative about what you can or cannot do now; your current tools and constraints are defined above." Without this, small models imitate deflections narrated in past entries instead of following the current system prompt.
- **Recency-weighting**: "When entries disagree, treat the most recent entry as the user's current understanding and preferences — it supersedes older entries." This prevents stale diary facts from overriding more recent corrections.
- Append `Tools:` with the dynamically generated tool descriptions (including configured MCP servers, if any) and guidance for preferring real data over shell commands.
- Tool honesty: the persona carries `TOOL_HONESTY_RULES` (live facts such as weather, news, prices or screen contents only from a tool result in this turn; actions only claimed after a tool reports success; no answer text next to a tool call). `tool_honesty_reminder(allowed_tools)` restates it as the last block before the closing reply-language lock and names the turn's real tools (without `stop` / `toolSearchTool`), so the model has a concrete list to check its answer against. Both are language- and tool-set-agnostic.
- Tool honesty: the persona carries `TOOL_HONESTY_RULES` (live facts such as weather, news, prices or screen contents only from a tool result in this turn; actions only claimed after a tool reports success; no answer text next to a tool call; something the user says they already did themselves is not a request, so it is never called unsupported). `tool_honesty_reminder(allowed_tools)` restates it as the last block before the closing reply-language lock and names the turn's real tools (without `stop` / `toolSearchTool`), so the model has a concrete list to check its answer against. Both are language- and tool-set-agnostic.
6. Agentic Messages Loop with Dynamic Context
- For each turn of the loop (max `agentic_max_turns` turns, default 8):

View File

@@ -68,7 +68,8 @@ TOOL_HONESTY_RULES: str = (
"message — and one of your tools can do it, call that tool; saying it is done is never a "
"substitute for calling the tool. Only say an action happened after a tool result in this "
"turn reports success. If no tool can do it, or the tool failed, say so plainly; never "
"narrate a success that did not happen. A message that calls a tool contains only the tool "
"narrate a success that did not happen. Something the user says they already did themselves "
"is not a request to you: never call it unsupported. A message that calls a tool contains only the tool "
"call: write no answer or confirmation next to it, and answer after you see the result."
)
@@ -248,7 +249,10 @@ def tool_honesty_reminder(tool_names) -> str:
f"must come from a tool call in this turn. Your tools this turn: {listed}. If one of "
"them can fetch the fact or do the action, call it now instead of answering. If none "
"of them can, say plainly that you cannot check or do that; never say something was "
"done unless a tool result in this turn says so."
"done unless a tool result in this turn says so. This is only about what the user asks "
"you to do now: something the user says they already did themselves is not a request, so "
"never call it unsupported or impossible; just do what they ask of you now, such as "
"talking or answering."
)

View File

@@ -54,12 +54,6 @@ TARGET_DB = -20.0 # loudness every speaker is brought to, dBFS RMS over the
MAX_GAIN_DB = 40.0 # a whisper at -60 dBFS still reaches the target; pure hiss is not blown up
MIN_GAIN_DB = -20.0
HIGHPASS_HZ = 60.0 # mains hum and desk rumble carry no speech
# Consonants ("ㅈ", "ㅅ") and soft sentence ends live above 1 kHz and are what a dark microphone or a
# swallowed syllable loses. Raised by a shelf that rises over one octave around HIGH_SHELF_HZ. Measured
# with large-v3-turbo (voiced masks from Silero): FLEURS Korean test split (382 clips, held out) 5.11% ->
# 4.85% CER, 25 live utterances of a dark speaker 5.04% -> 3.92%; 6 dB did little, 18-24 dB no better.
HIGH_SHELF_HZ = 1000.0
HIGH_SHELF_DB = 12.0
_KNEE = 0.8 # soft limiter: peaks above this bend smoothly towards the ceiling instead of clipping
_CEILING = 0.98
_WINDOW = 512
@@ -76,10 +70,7 @@ def loudness_equalise(x: np.ndarray, voiced: np.ndarray | None = None) -> np.nda
if len(x) == 0:
return x
spec = np.fft.rfft(x.astype(np.float64))
freqs = np.fft.rfftfreq(len(x), 1 / TARGET_RATE)
spec[freqs < HIGHPASS_HZ] = 0
rise = np.clip(np.log2(np.maximum(freqs, 1.0) / HIGH_SHELF_HZ) + 0.5, 0.0, 1.0)
spec *= 10 ** (HIGH_SHELF_DB * rise / 20)
spec[np.fft.rfftfreq(len(x), 1 / TARGET_RATE) < HIGHPASS_HZ] = 0
y = np.fft.irfft(spec, len(x))
n_win = -(-len(y) // _WINDOW)
frames = np.pad(y, (0, n_win * _WINDOW - len(y))).reshape(n_win, _WINDOW)

View File

@@ -108,18 +108,9 @@ its own streaming Silero VAD (512-sample windows, state kept between frames).
the gain kept between -20 and +40 dB, and peaks above 0.8 are soft-limited below 0.98 instead of
clipping. Measured: speech from -60 dBFS to clipped shouting was transcribed equally well, and quiet
speech under loud clicks or knocks better than with peak normalisation.
Before the level is measured, the band where consonants such as "ㅈ" and "ㅅ" live is raised by
12 dB: a shelf that rises over one octave from 707 Hz to 1.41 kHz (centred on 1 kHz) and stays
flat above. A dark microphone, or a speaker who swallows consonants and sentence ends, loses that
band first. Measured with large-v3-turbo, each clip levelled with its Silero voiced windows as a
final is: on 32 real Discord utterances of such a speaker, 5.04% character errors (18 of 357) on
the 25 whose words are sure, down to 3.92% (14), with sentence ends ("어때", "틀어줘") and words
("연제구는") recovered and none lost; on FLEURS Korean, checked on the held-out test split (382
clips), 5.11% to 4.85%. A wake word whose first consonant is not heard at all ("자비스" as
"아비스" or "다비스") is not fixed by it (14 of 23 exact before and after); call detection's near
match catches those one letter away. 6 dB changed little; 18 and 24 dB were no better on FLEURS
and worse on the real speaker. The noise check below is unchanged with it (108 clips, none with
text).
No treble boost: a 12 dB shelf above 1 kHz (0.1.34) helped FLEURS a little but, on the user's
own 34 live utterances of 2026-11-18, turned "왜" into "올해" and "알아서" into "아라해석" and
dropped a whole sentence, so it was removed again.
### Scheduling

View File

@@ -5,7 +5,7 @@ from __future__ import annotations
import numpy as np
import pytest
from voice_worker.audio import HIGH_SHELF_DB, HIGH_SHELF_HZ, InputConverter, loudness_equalise
from voice_worker.audio import InputConverter, loudness_equalise
from voice_worker.protocol import (KIND_AUDIO, KIND_TTS, ProtocolError, decode_audio_frame, encode_audio_frame,
encode_tts_frame, decode_tts_frame)
@@ -171,16 +171,3 @@ def test_low_rumble_is_removed_before_levelling():
spec = np.abs(np.fft.rfft(y))
freqs = np.fft.rfftfreq(len(y), 1 / 16000)
assert spec[np.argmin(np.abs(freqs - 30))] < 0.01 * spec[np.argmin(np.abs(freqs - 220))]
@pytest.mark.unit
def test_consonant_band_is_raised_against_the_vowel_band():
"""A dark microphone (or a speaker who swallows consonants) loses the band above 1 kHz where
"ㅈ" and "ㅅ" live; it is raised by the shelf, the band below it is left as it was."""
t = np.arange(32000) / 16000
low, high = HIGH_SHELF_HZ / 4, HIGH_SHELF_HZ * 4
x = (0.05 * np.sin(2 * np.pi * low * t) + 0.05 * np.sin(2 * np.pi * high * t)).astype(np.float32)
spec = np.abs(np.fft.rfft(loudness_equalise(x)))
freqs = np.fft.rfftfreq(len(x), 1 / 16000)
ratio_db = 20 * np.log10(spec[np.argmin(np.abs(freqs - high))] / spec[np.argmin(np.abs(freqs - low))])
assert abs(ratio_db - HIGH_SHELF_DB) < 0.5