Compare commits

...

12 Commits

Author SHA1 Message Date
javis-bot
478968949c chore(app): version 0.1.47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:42:02 +09:00
javis-bot
caad09c613 feat(conversation): what is said in pieces is answered once, and a dropped answer stops the brain
Live 2026-11-19 every piece of one thought (pauses of 0.1-4.7 s) was answered on its own and the
brain wrote each cancelled answer to its end, so the replies queued up to 35 s.

- conversation: a speaker who goes on before their answer is heard (within call.continueMs,
  default 5 s) has the earlier request taken back (policy withdraw, not missed) and the joined
  text answered as one request; a stop still stops; no second earcon; partials prepare the joined
  text
- voice worker: a reply cancelled while waiting is never started, a running one stops at the
  engine's next model or tool call (turn_state.cancel), and it uses up no turn, so a confirmation
  carried by the joined request does not lapse
- specs: conversation.spec.md (Going on), voice_worker.spec.md, llm_contexts.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:41:50 +09:00
javis-bot
97b827f424 docs(v2): STT accuracy plan, hypotheses, test sets and decision rules before any change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:23:18 +09:00
javis-bot
045218b008 docs(v2): roadmap M8 clean Windows install to first conversation checked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:35:56 +09:00
javis-bot
e32671f871 chore(app): version 0.1.46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:01:40 +09:00
javis-bot
e23a962923 feat(app): side menu in groups that open, settings split into tidy screens
The side menu keeps 대시보드 and 대화 on top and puts the rest in groups that
open when their name is pressed (디스코드, 대화 방식, 음성·모델, 기능, PC, 관리);
the shown screen's group opens and open groups are kept. 채널·호출 is split
into 음성 채널, 호출 and 여럿이 말할 때, and 모델 into 받아 적기·말하기 and 대화
모델; the chat wake words join 호출, the recording switch 음성 채널 and the
default location 기능 켜기·MCP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:01:16 +09:00
javis-bot
716b5bb2e6 feat(app): Jarvis is called like a secretary by default
The default wake words are 비서, 비서야, 비서님, 김비서, 김비서야 and 김비서님.
Where several wake words are heard starting at the same place, the closest
and then the longest is the call, so a call form is removed whole (비서야 멈춰
is a stop) and a near match of a longer word does not eat into what follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:56:38 +09:00
javis-bot
e88c94d934 docs(v2): roadmap M7 screen share viewing, with the eval numbers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:52:31 +09:00
javis-bot
4906d8e7dd chore(app): version 0.1.45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:49:25 +09:00
javis-bot
844958d311 feat(v2): screenView sees someone's Discord screen share in its popped-out window
The share target checks that the person asked about shares their screen in
Jarvis's voice channel, finds Discord's popped-out window (title not ending in
Discord, the voice channel's name first) and, when there is none, asks for it in
the channel and looks again every second for 30 s. Screen actions now wait as
long as PC actions in the worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:43:31 +09:00
javis-bot
3af881935c chore(app): version 0.1.44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
javis-bot
20bd90f65e fix(v2): pcControl keys to a minimised window bring back the window that was in front
restore activates the window (SW_RESTORE), so restoring before keys made the
window already in front: the window that was in front before was not brought
back, and the game / full-screen check came after the window had moved. Keys
now note the window in front and check for a game first, then restore inside
the moment in front. A minimised window never takes keys behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:08:30 +09:00
39 changed files with 1081 additions and 177 deletions

View File

@@ -18,7 +18,7 @@
- **빠른 음성 대화**: 말하는 도중에 전사하고 답은 문장 단위로 바로 읽어서, 말이 끝난 뒤 첫 소리까지 약 1~1.5초를 목표로 합니다.
- **최대 10명 동시 대화**: 화자별로 따로 받아 전사합니다. 여러 명이 동시에 부를 때 어떻게 답할지(처음 대화만 / 들어온 순서대로 / 마지막 대화만 / 제일 중요한 대화만)는 GUI에서 고릅니다.
- **호출어와 지정한 사람**:
- 호출어는 GUI에서 여러 개 등록합니다.
- 호출어는 GUI에서 여러 개 등록합니다. 기본값은 "비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"입니다.
- "지정한 사람만 응답"과 "지정한 사람은 호출어 없이 대답"을 각각 켜고 끕니다.
- **AI 채팅 채널**: 등록한 디스코드 채팅 채널을 자비스가 읽고, 메시지를 보내고, 자기 메시지를 고칩니다. 채널 ID, 이름, 설명은 GUI에서 편집하고, 고친 이름·설명은 실제 디스코드 채널 이름·주제에도 반영합니다(끌 수 있음).
- **예약 알림과 정기 작업**: "10분 뒤에 알려 줘", "내일 7시에 회의 알려 줘", "평일 아침 9시에 날씨 알려 줘"처럼 말이나 채팅으로 맡깁니다. 맡긴 사람이 자비스와 같은 음성 채널에 있으면 음성으로, 아니면 채팅 채널에 글로 전합니다. 정기 작업은 날씨, 웹 검색처럼 찾아보는 기능만 쓰고, 관리 앱 "예약" 화면에서 목록을 보고 지울 수 있습니다.

2
app/Cargo.lock generated
View File

@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.43"
version = "0.1.47"
dependencies = [
"async-trait",
"base64 0.22.1",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-app"
version = "0.1.43"
version = "0.1.47"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"

4
app/shell/Cargo.lock generated
View File

@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.43"
version = "0.1.47"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -2666,7 +2666,7 @@ dependencies = [
[[package]]
name = "jarvis-shell"
version = "0.1.43"
version = "0.1.47"
dependencies = [
"jarvis-app",
"serde_json",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-shell"
version = "0.1.43"
version = "0.1.47"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"

View File

@@ -194,6 +194,21 @@ one window; the app captures and decides who may.
`window` takes the visible top-level window whose title or program name contains the text
(case-insensitive; the one nearest the front when several do), also when others cover it; a
minimised window is refused ("최소화되어 있어 볼 수 없습니다").
- **Screen share** (`share`, `docs/v2/pc-features.md` "다른 사람의 디스코드 화면공유 보기"):
someone's Discord screen share, seen in the window Discord pops it out to; no account or token is
touched and Jarvis never clicks Discord. Only while Jarvis is in a voice channel ("자비스가 음성
채널에 있을 때만 화면공유를 볼 수 있습니다") and someone there other than Jarvis shares their
screen (`self_stream`): with a `person`, the sharer whose display name contains it or is contained
in it (letters and digits, any case), else "<person>님은 지금 화면공유 중이 아닙니다"; with none,
anyone sharing ("지금 음성 채널에서 화면공유 중인 사람이 없습니다" when nobody does). The window
is a visible Discord window (`discord.exe`, `discordptb.exe`, `discordcanary.exe`) whose title does
not end with "Discord" (the main window's does), the one whose title contains the voice channel's
name first, else the one nearest the front; Discord on the exclusion list is never seen. A
minimised one is refused ("창을 다시 열어 주세요"). With none, a voice request says "디스코드에서
<name>님 화면공유를 시청하고 창 분리 버튼을 눌러 주세요." in the channel (without a name when
several share) and looks again every second for 30 s, then refuses ("화면공유 창이 분리되지 않아
볼 수 없습니다"); a chat request is refused at once with that instruction. Who may see is decided
before anything is said. The worker waits 45 s for a screen action ("PC control" timing).
- **Exclusions**: `pcControl.screenExclude` (program file names, ".exe" may be left out; default
`1Password`, `Bitwarden`, `KeePass`, `KeePassXC`). Their windows are never captured or listed, and
in a monitor or whole-screen picture their area is painted black before anything leaves the
@@ -212,8 +227,8 @@ one window; the app captures and decides who may.
saved, never logged.
- **List**: monitors `{number, width, height, main}` and visible windows `{program, title,
minimised}`, front first, at most 30, without excluded programs and the app's own windows.
- **Activity**: each capture is `screen.view` with the target (`전체 화면`, `모니터 <n>`, `창`; never
a title), who asked and whether it worked ("Activity").
- **Activity**: each capture is `screen.view` with the target (`전체 화면`, `모니터 <n>`, `창`,
`화면공유`; never a title or a name), who asked and whether it worked ("Activity").
- Logs carry the action, target kind, size and time taken, never a title or a picture.
## PC control (`app/src/pc.rs`)
@@ -254,7 +269,9 @@ agent-desktop and decides who may.
has none; win-ci VM 2026-11-19), so keys go behind only where there is no doubt where they land.
Store and WinUI apps (the new Calculator), Chromium, Electron and Firefox windows have none.
- **In front for a moment**: a combination, any other key, and every key for a window that does
not take keys behind. The window is brought to the front (`focus-window`, then `press`) and the
not take keys behind or is minimised. A minimised window is restored inside the moment
(restoring brings it forward, so the window in front is noted before and brought back after;
the window itself stays restored). The window is brought to the front (`focus-window`, then `press`) and the
window that was in front before is brought back right after (`focus-window` on it, also when
`press` failed; one that cannot be brought back is left, with a warning in the log). For a
window whose process agent-desktop cannot find from the window (the new Calculator, a Store
@@ -315,7 +332,8 @@ agent-desktop and decides who may.
interactive elements read afresh after the action (at most 200), so the brain says it was done
only from what it sees; an action that closes its window (a dialog's button, Enter) answers the
window with no elements. agent-desktop's error codes come back as Korean reasons (`STALE_REF`: "화면이
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting).
바뀌었습니다. 다시 읽어 주세요"; a minimised window is restored before acting; for `keys` inside
the moment in front, "Keys").
- **Refused before asking**: the stop, the anti-cheat, an administrator window (a process whose
elevation cannot be read counts as one) and a password field are refused without asking.
`windows` and `read` at `ask` answer `ask: true` like the rest.
@@ -867,8 +885,13 @@ its data folder, and again after each one is saved. The log hub writes to the da
## Screens (`app/ui/`)
Plain HTML, CSS and an ES module, no build step, dark theme after `DESIGN.md`. All text is Korean.
The side menu has 대화 right under 대시보드 and 앱 last; there is no separate update screen (updates
are on the 앱 screen) and no separate 한 일 screen (it is a view of the 로그 screen).
The side menu has 대시보드 and 대화 on top, then groups that open and close when their name is
pressed (user request, 2026-11-19): 디스코드 (봇, 음성 채널, 채팅 채널), 대화 방식 (호출, 여럿이 말할
때, 대화 기억, 사람별 지침), 음성·모델 (받아 적기·말하기, 대화 모델), 기능 (기능 켜기·MCP, 예약,
백그라운드 수집), PC (PC 제어·화면 보기, 게임 모드) and 관리 (로그, 앱). The group of the screen
shown opens (and the 관리 group when a new version renames the 앱 button); which groups are open is
kept in the page's storage for the next start. There is no separate update screen (updates are on
the 앱 screen) and no separate 한 일 screen (it is a view of the 로그 screen).
- **First-run wizard** (while `firstRun`): data folder (free space, warning under 200 GB), PC check,
the model to download, Discord bot (token check with the
@@ -881,7 +904,7 @@ are on the 앱 screen) and no separate 한 일 screen (it is a view of the 로
Strata's last line, and stop in the header; the header with start, restart and stop is rebuilt
only when the state changes, so a click is never lost to the refresh. Every screen change starts
a new generation: an answer that arrives after the user moved to another screen is dropped.
- **Settings screens** (봇, 채널·호출, 채팅 채널, 대화 기억, 모델, 기능, 백그라운드 수집, 예약, PC 제어, 게임 모드, 앱): built from `settings_fields`; only
- **Settings screens** (봇, 음성 채널, 호출, 여럿이 말할 때, 채팅 채널, 대화 기억, 받아 적기·말하기, 대화 모델, 기능 켜기·MCP, 백그라운드 수집, 예약, PC 제어·화면 보기, 게임 모드, 앱): built from `settings_fields`, each field on one screen: 음성 채널 has the server, the auto-join channel, the speakers, bots, the join notice and the recording switch; 호출 the voice and chat wake words and `call.*`; 여럿이 말할 때 `response.*`; 받아 적기·말하기 `voice.stt.*` and the speech speed; 대화 모델 `llm.*`; the default location is on 기능 켜기·MCP (weather and search use it); only
changed fields are sent; the server and voice channel are pickers (choosing another server clears
the voice channel and lists only that server's channels); the chat channels are a table
with their switches; the bot screen also changes the token; the 기능 screen has the feature

View File

@@ -31,6 +31,10 @@ pub struct CallSettings {
/// After a wake word said alone, how long the speaker's next utterance may take to start and
/// still be joined to it (the pause after "Jarvis," often ends an utterance).
pub follow_up: Duration,
/// A speaker who goes on talking before Jarvis's answer to them is heard, starting within this of
/// their last utterance's end, is still asking: what they say is joined to the request
/// (`call.continueMs`; zero turns it off).
pub continue_window: Duration,
/// Play the earcon when Jarvis takes a voice call (`call.earcon`).
pub earcon: bool,
}
@@ -38,7 +42,7 @@ pub struct CallSettings {
impl Default for CallSettings {
fn default() -> Self {
Self {
wake_words: vec!["자비스".into()],
wake_words: crate::settings::default_wake_words(),
similarity: 0.2,
mode: CallMode::WakeWord,
designated: HashSet::new(),
@@ -46,6 +50,7 @@ impl Default for CallSettings {
designated_no_wake_word: false,
stop_words: ["멈춰", "그만", "정지", "stop"].map(String::from).to_vec(),
follow_up: Duration::from_millis(1500),
continue_window: Duration::from_millis(5000),
earcon: true,
}
}
@@ -70,9 +75,9 @@ pub fn normalise(text: &str) -> Vec<char> {
text.nfkd().flat_map(char::to_lowercase).filter(|c| c.is_alphanumeric()).collect()
}
/// The stretch `[start, end)` of `text` closest to `word`, if within `max_edits` edits
/// (approximate substring matching: the match may start and end anywhere in `text`).
fn find(text: &[char], word: &[char], max_edits: usize) -> Option<(usize, usize)> {
/// The stretch `[start, end)` of `text` closest to `word` and its edits, if within `max_edits`
/// edits (approximate substring matching: the match may start and end anywhere in `text`).
fn find(text: &[char], word: &[char], max_edits: usize) -> Option<(usize, usize, usize)> {
if word.is_empty() || text.is_empty() {
return None;
}
@@ -104,7 +109,7 @@ fn find(text: &[char], word: &[char], max_edits: usize) -> Option<(usize, usize)
(1..=n)
.filter(|&j| d[m][j] <= max_edits)
.min_by_key(|&j| (d[m][j], j))
.map(|j| (start[m][j], j))
.map(|j| (start[m][j], j, d[m][j]))
}
impl CallSettings {
@@ -114,13 +119,15 @@ impl CallSettings {
/// The text with the first heard wake word removed, if one is heard.
fn without_wake_word(&self, text: &[char]) -> Option<Vec<char>> {
// The earliest heard wake word is the one used to address Jarvis.
let (s, e) = self
// The earliest heard wake word is the one used to address Jarvis; where several start
// there, the closest, then the longest ("비서야" over "비서"), so no part of the call is
// left over and a near match does not eat into what follows.
let (s, e, _) = self
.wake_words
.iter()
.map(|w| normalise(w))
.filter_map(|word| find(text, &word, self.allowed_edits(&word)))
.min()?;
.min_by_key(|&(s, e, edits)| (s, edits, std::cmp::Reverse(e)))?;
Some(text[..s].iter().chain(&text[e..]).copied().collect())
}
@@ -169,8 +176,9 @@ mod tests {
const A: u64 = 1;
const B: u64 = 2;
/// The mechanism tests use one wake word of their own, whatever the default is.
fn s() -> CallSettings {
CallSettings::default()
CallSettings { wake_words: vec!["자비스".into()], ..CallSettings::default() }
}
#[test]
@@ -264,4 +272,25 @@ mod tests {
assert_eq!(c.judge(A, "javis 날씨"), Call::Request { direct: true });
assert_eq!(c.judge(A, "javis stop"), Call::Stop);
}
#[test]
fn the_longest_wake_word_where_the_call_starts_is_the_one_removed() {
let c = CallSettings { wake_words: vec!["비서".into(), "비서야".into(), "김비서".into()], ..s() };
assert!(c.wake_word_only("비서야,"), "the call form is the whole wake word, not 비서 and a 야 left over");
assert!(c.wake_word_only("김비서?"));
assert_eq!(c.judge(A, "비서야 멈춰"), Call::Stop);
assert_eq!(c.judge(A, "김비서 그만해"), Call::Stop);
}
#[test]
fn by_default_jarvis_is_called_like_a_secretary() {
let c = CallSettings::default();
for call in ["비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"] {
assert!(c.wake_word_only(&format!("{call},")), "{call}");
assert_eq!(c.judge(A, &format!("{call} 오늘 날씨 어때?")), Call::Request { direct: true }, "{call}");
assert_eq!(c.judge(A, &format!("{call} 멈춰")), Call::Stop, "{call}");
}
assert_eq!(c.judge(A, "오늘 날씨 어때?"), Call::Ignore);
assert_eq!(c.judge(A, "자비스 날씨"), Call::Ignore, "the old name no longer calls");
}
}

View File

@@ -167,6 +167,18 @@ pub struct Conversation<V: Voice> {
wakes: HashMap<u64, (Request, Option<Instant>)>,
/// Requests whose earcon already played (a wake word said alone was answered with it).
chimed: HashSet<u64>,
/// Each speaker's latest voice request whose answer has not been heard yet: what they say next
/// may still be part of it ("Going on").
turns: HashMap<u64, Turn>,
}
/// A voice request whose answer is not heard yet.
struct Turn {
id: u64,
text: String,
ended_at: Instant,
/// Still a "was it said to Jarvis?" question (its ask id), not yet a request of the policy.
ask: Option<String>,
}
impl<V: Voice> Conversation<V> {
@@ -185,7 +197,7 @@ impl<V: Voice> Conversation<V> {
Self { voice, calls, policy: Policy::new(policy), next_id: 0, answers: HashMap::new(),
by_speech: HashMap::new(), asks: HashMap::new(), missed: MissedLog::default(), chat: ChatCalls::default(),
chat_answers: HashMap::new(), started: HashMap::new(), wakes: HashMap::new(),
chimed: HashSet::new() }
chimed: HashSet::new(), turns: HashMap::new() }
}
/// Answer the registered chat channels too.
@@ -225,6 +237,13 @@ impl<V: Voice> Conversation<V> {
/// would be) lets the brain get ready while the speaker is still talking. Not while Jarvis is
/// answering: the call may not be taken, and the brain is busy with the answer.
fn on_partial(&mut self, speaker: u64, partial: &str) {
// Going on with a request whose answer is not heard yet: it will be joined and taken.
if let Some(t) = self.turns.get(&speaker) {
if !self.calls.continue_window.is_zero() && !matches!(self.calls.judge(speaker, partial), Call::Stop) {
self.voice.prepare(speaker, &format!("{} {}", t.text, partial));
}
return;
}
if !self.answers.is_empty() {
return;
}
@@ -277,10 +296,15 @@ impl<V: Voice> Conversation<V> {
self.started.remove(&speaker);
}
// What follows a wake word said alone is joined to it.
let (text, chimed) = match self.wakes.remove(&speaker) {
let (mut text, mut chimed) = match self.wakes.remove(&speaker) {
Some((wake, _)) => (format!("{} {}", wake.text, f.text), self.chimed.remove(&wake.id)),
None => (f.text.clone(), false),
};
// The speaker goes on before their answer was heard: still the same request.
if let Some((earlier, earcon_played)) = self.going_on(speaker, f, now) {
text = format!("{earlier} {text}");
chimed |= earcon_played;
}
self.next_id += 1;
if chimed {
self.chimed.insert(self.next_id);
@@ -316,21 +340,19 @@ impl<V: Voice> Conversation<V> {
let a = self.policy.utterance_closed(speaker, now);
self.apply(a, now);
}
Call::Request { direct } => {
let a = self.policy.request(Request { direct, ..req }, now);
self.apply(a, now);
}
Call::Request { direct } => self.voice_request(Request { direct, ..req }, now),
Call::Ask => {
let room = self.room(speaker);
if room.people <= 1 {
// Alone with Jarvis: whatever the speaker says is said to it.
debug_log!("conversation", "addressed req={} alone with jarvis", req.id);
let a = self.policy.request(req, now);
self.apply(a, now);
self.voice_request(req, now);
return;
}
let id = format!("q{}", req.id);
if self.voice.ask_addressed(&id, speaker, &text, room) {
self.turns.insert(speaker, Turn { id: req.id, text: req.text.clone(),
ended_at: req.ended_at, ask: Some(id.clone()) });
self.asks.insert(id, (req, now));
} else {
let a = self.policy.utterance_closed(speaker, now);
@@ -344,12 +366,13 @@ impl<V: Voice> Conversation<V> {
// An answer after the timeout counts as no, even before the timer ran.
let yes = *addressed && now < asked + ADDRESSED_TIMEOUT;
debug_log!("conversation", "addressed req={} yes={yes}", req.id);
let a = if yes {
self.policy.request(req, now)
self.turns.retain(|_, t| t.id != req.id);
if yes {
self.voice_request(req, now);
} else {
self.policy.utterance_closed(req.speaker, now)
};
self.apply(a, now);
let a = self.policy.utterance_closed(req.speaker, now);
self.apply(a, now);
}
}
}
CoreEvent::Ranked { id, order } => {
@@ -358,9 +381,16 @@ impl<V: Voice> Conversation<V> {
let a = self.policy.ranked(rank_id, &order, now);
self.apply(a, now);
}
// The answer is heard: what its asker says next is a new request.
CoreEvent::SpeakStart(id) => {
if let Some(pid) = self.by_speech.get(id).copied() {
self.turns.retain(|_, t| t.id != pid);
}
}
CoreEvent::SpeakEnd(e) => {
if let Some(pid) = self.by_speech.remove(&e.id) {
self.answers.remove(&pid);
self.turns.retain(|_, t| t.id != pid);
debug_log!("conversation", "answer req={pid} done played={} cancelled={} failed={}",
e.played, e.cancelled, e.failed);
let a = self.policy.answer_done(pid, now);
@@ -378,6 +408,7 @@ impl<V: Voice> Conversation<V> {
}
CoreEvent::WorkerLost => {
self.started.clear();
self.turns.clear();
for (_, (req, _)) in self.wakes.drain() {
self.chimed.remove(&req.id);
}
@@ -386,9 +417,42 @@ impl<V: Voice> Conversation<V> {
}
}
/// A voice request goes to the policy; until its answer is heard, its speaker may go on.
fn voice_request(&mut self, req: Request, now: Instant) {
self.turns.insert(req.speaker, Turn { id: req.id, text: req.text.clone(), ended_at: req.ended_at, ask: None });
let a = self.policy.request(req, now);
self.apply(a, now);
}
/// The final `f` continues the speaker's request whose answer is not heard yet (it started within
/// `call.continueMs` of that request's end, and is not a stop): the request is taken back, its
/// running answer stopped. Returns its text and whether its earcon already played.
fn going_on(&mut self, speaker: u64, f: &crate::discord::HeardFinal, now: Instant) -> Option<(String, bool)> {
let t = self.turns.get(&speaker)?;
let started = now.checked_sub(Duration::from_millis(f.wait_ms + f.audio_ms)).unwrap_or(now);
if self.calls.continue_window.is_zero() || started > t.ended_at + self.calls.continue_window
|| matches!(self.calls.judge(speaker, &f.text), Call::Stop) {
return None;
}
let t = self.turns.remove(&speaker).expect("checked");
let earcon_played = self.chimed.remove(&t.id) || (self.calls.earcon && self.answers.contains_key(&t.id));
debug_log!("conversation", "going on req={} speaker={speaker}", t.id);
match t.ask {
Some(q) => {
self.asks.remove(&q);
}
None => {
let a = self.policy.withdraw(t.id);
self.apply(a, now);
}
}
Some((t.text, earcon_played))
}
/// The policy's stop, and every pending "was it said to Jarvis?" question is void.
fn stop_everything(&mut self, now: Instant) {
self.wakes.clear();
self.turns.clear();
let a = self.policy.stop();
self.apply(a, now);
self.chimed.clear(); // nothing is waiting any more
@@ -405,6 +469,7 @@ impl<V: Voice> Conversation<V> {
for id in late {
if let Some((req, _)) = self.asks.remove(&id) {
debug_log!("conversation", "addressed req={} timed out", req.id);
self.turns.retain(|_, t| t.id != req.id);
let a = self.policy.utterance_closed(req.speaker, now);
self.apply(a, now);
}
@@ -415,8 +480,7 @@ impl<V: Voice> Conversation<V> {
for speaker in due {
if let Some((req, _)) = self.wakes.remove(&speaker) {
debug_log!("conversation", "wake word alone req={} answered alone", req.id);
let a = self.policy.request(req, now);
self.apply(a, now);
self.voice_request(req, now);
}
}
let a = self.policy.tick(now);
@@ -457,6 +521,7 @@ impl<V: Voice> Conversation<V> {
}
Action::Missed(r, reason) => {
self.chimed.remove(&r.id);
self.turns.retain(|_, t| t.id != r.id);
debug_log!("conversation", "missed req={} speaker={} reason={reason:?}", r.id, r.speaker);
let mut missed = self.missed.lock().unwrap();
if missed.len() == MISSED_KEPT {
@@ -502,6 +567,11 @@ pub async fn run<V: Voice>(mut conv: Conversation<V>, mut events: mpsc::Unbounde
#[cfg(test)]
mod tests {
use super::*;
/// The tests call Jarvis by one wake word of their own, whatever the default is.
fn test_calls() -> CallSettings {
CallSettings { wake_words: vec!["자비스".into()], ..CallSettings::default() }
}
use crate::discord::{Heard, HeardFinal};
use crate::playback::PlaybackEnd;
use crate::policy::PolicyKind;
@@ -585,10 +655,11 @@ mod tests {
const A: u64 = 11;
const B: u64 = 22;
const C: u64 = 33;
fn conv(kind: PolicyKind, mode: CallMode) -> (Conversation<Fake>, Fake, Instant) {
let f = Fake::default();
let calls = CallSettings { mode, earcon: false, ..CallSettings::default() };
let calls = CallSettings { mode, earcon: false, ..test_calls() };
let policy = PolicySettings { kind, collect: Duration::ZERO, ..PolicySettings::default() };
*f.people.borrow_mut() = vec![A, B];
(Conversation::new(f.clone(), calls, policy), f, Instant::now())
@@ -596,7 +667,7 @@ mod tests {
fn conv_earcon(kind: PolicyKind) -> (Conversation<Fake>, Fake, Instant) {
let f = Fake::default();
let policy = PolicySettings { kind, collect: Duration::ZERO, ..PolicySettings::default() };
(Conversation::new(f.clone(), CallSettings::default(), policy), f, Instant::now())
(Conversation::new(f.clone(), test_calls(), policy), f, Instant::now())
}
#[test]
@@ -608,7 +679,7 @@ mod tests {
assert!(f.take().is_empty(), "a waiting request gets its earcon when its answer starts");
c.on_event(&end("s1", false), ms(t, 3000));
assert_eq!(f.take(), vec![Did::Earcon, Did::Answer(B, "자비스 불 꺼 줘".into())]);
c.on_event(&fin(B, "오늘 뭐 먹지", 100), ms(t, 3100));
c.on_event(&fin(C, "오늘 뭐 먹지", 100), ms(t, 3100));
assert!(f.take().is_empty(), "no call, no earcon");
}
@@ -624,7 +695,7 @@ mod tests {
// Nothing followed the next one: it is answered alone, still with one earcon.
c.on_event(&end("s1", false), ms(t, 5000));
c.on_event(&fin_utt(A, 3, "자비스"), ms(t, 6000));
c.tick(ms(t, 6000) + CallSettings::default().follow_up);
c.tick(ms(t, 6000) + test_calls().follow_up);
assert_eq!(f.take(), vec![Did::Earcon, Did::Answer(A, "자비스".into())]);
}
@@ -654,10 +725,10 @@ mod tests {
c.on_event(&fin_utt(B, 1, "자비스"), ms(t, 100));
c.on_event(&partial(B, "지금 몇 시"), ms(t, 300));
assert_eq!(f.take(), vec![Did::Prepare(B, "자비스 지금 몇 시".into())]);
// Not while an answer runs.
// Not while an answer runs (to someone else: its asker going on is joined to it).
c.on_event(&fin(A, "자비스 날씨", 100), ms(t, 400));
f.take();
c.on_event(&partial(A, "자비스 내일은"), ms(t, 500));
c.on_event(&partial(C, "자비스 내일은"), ms(t, 500));
assert!(f.take().is_empty());
}
@@ -708,7 +779,7 @@ mod tests {
c.on_event(&start(A, 1), ms(t, 0));
c.on_event(&start(A, 2), ms(t, 600)); // the final of utterance 1 comes after 2 started
c.on_event(&fin_utt(A, 1, "자비스"), ms(t, 900));
c.tick(ms(t, 900) + CallSettings::default().follow_up + Duration::from_millis(1));
c.tick(ms(t, 900) + test_calls().follow_up + Duration::from_millis(1));
assert!(f.take().is_empty());
c.on_event(&fin_utt(A, 2, "멈춰"), ms(t, 3000));
assert_eq!(f.take(), vec![Did::StopAll], "joined, it is a stop");
@@ -719,7 +790,7 @@ mod tests {
let (mut c, f, t) = conv(PolicyKind::First, CallMode::WakeWord);
c.on_event(&start(A, 1), ms(t, 0));
c.on_event(&fin_utt(A, 1, "자비스"), ms(t, 900));
let wait = CallSettings::default().follow_up;
let wait = test_calls().follow_up;
assert_eq!(c.next_deadline(), Some(ms(t, 900) + wait));
c.tick(ms(t, 900) + wait - Duration::from_millis(1));
assert!(f.take().is_empty());
@@ -748,7 +819,7 @@ mod tests {
assert!(f.take().is_empty(), "no wake word: not a call");
c.on_event(&start(A, 2), ms(t, 62_000));
c.on_event(&fin_utt(A, 2, "자비스"), ms(t, 63_000));
let wait = CallSettings::default().follow_up;
let wait = test_calls().follow_up;
assert_eq!(c.next_deadline(), Some(ms(t, 63_000) + wait));
c.tick(ms(t, 63_000) + wait);
assert_eq!(f.take(), vec![Did::Answer(A, "자비스".into())]);
@@ -759,7 +830,7 @@ mod tests {
let (mut c, f, t) = conv(PolicyKind::First, CallMode::WakeWord);
c.on_event(&fin_utt(A, 1, "자비스"), ms(t, 900));
c.on_event(&CoreEvent::Stopped, ms(t, 1000));
c.tick(ms(t, 900) + CallSettings::default().follow_up);
c.tick(ms(t, 900) + test_calls().follow_up);
assert!(f.take().is_empty());
assert_eq!(c.next_deadline(), None);
}
@@ -826,7 +897,7 @@ mod tests {
let f = Fake::default();
*f.people.borrow_mut() = vec![A, B, 33];
let calls = CallSettings { mode: CallMode::Addressed, earcon: false, designated: [A].into(), designated_only: true,
..CallSettings::default() };
..test_calls() };
let mut c = Conversation::new(f.clone(), calls, PolicySettings { kind: PolicyKind::InOrder, ..PolicySettings::default() });
c.on_event(&fin(A, "불 좀 꺼 줄래", 100), ms(t, 0));
assert!(matches!(f.take()[..], [Did::Addressed(_, A)]));
@@ -899,7 +970,7 @@ mod tests {
let (mut c, f, t) = conv(PolicyKind::InOrder, CallMode::Addressed);
c.on_event(&fin(A, "자비스 날씨", 100), ms(t, 0));
c.on_event(&fin(B, "자비스 뉴스", 100), ms(t, 100));
c.on_event(&fin(B, "불 좀 꺼 줄래", 100), ms(t, 200)); // waiting for "addressed"
c.on_event(&fin(C, "불 좀 꺼 줄래", 100), ms(t, 200)); // waiting for "addressed"
let did = f.take();
let q = did.iter().find_map(|d| if let Did::Addressed(q, _) = d { Some(q.clone()) } else { None }).unwrap();
// The slash command stops all speech: Stopped comes before the speech's end.
@@ -930,7 +1001,7 @@ mod tests {
(CH_ALL, ChatChannelCall { everything: true, send: true }),
(CH_MUTE, ChatChannelCall { everything: true, send: false }),
].into() };
let voice = CallSettings { earcon: false, ..CallSettings::default() };
let voice = CallSettings { earcon: false, ..test_calls() };
let c = Conversation::new(f.clone(), voice, PolicySettings::default()).with_chat(chat);
(c, f, Instant::now())
}
@@ -942,7 +1013,7 @@ mod tests {
#[test]
fn a_chat_call_is_answered_as_a_reply_to_its_message() {
let (mut c, f, t) = chat_conv(CallSettings::default());
let (mut c, f, t) = chat_conv(test_calls());
c.on_event(&chat(CH_WAKE, 5, A, false, "자비스 내일 일정 알려 줘"), t);
let Did::ChatAnswer(id, user, text) = f.take().remove(0) else { panic!() };
assert_eq!((user, text.as_str()), (A, "자비스 내일 일정 알려 줘"));
@@ -956,7 +1027,7 @@ mod tests {
#[test]
fn chat_answers_follow_each_channels_switch_and_never_answer_bots() {
let (mut c, f, t) = chat_conv(CallSettings::default());
let (mut c, f, t) = chat_conv(test_calls());
c.on_event(&chat(CH_WAKE, 1, A, false, "오늘 뭐 먹지"), t);
assert!(f.take().is_empty(), "wake word only");
c.on_event(&chat(CH_ALL, 2, A, false, "오늘 뭐 먹지"), t);
@@ -974,7 +1045,7 @@ mod tests {
#[test]
fn chat_uses_its_own_wake_words_and_the_designated_people() {
let calls = CallSettings { wake_words: vec!["봇".into()], designated: [A].into(), designated_only: true,
..CallSettings::default() };
..test_calls() };
let (mut c, f, t) = chat_conv(calls);
c.on_event(&chat(CH_WAKE, 1, A, false, "봇 안녕"), t);
assert_eq!(f.take().len(), 1);
@@ -986,7 +1057,7 @@ mod tests {
#[test]
fn a_failed_chat_answer_posts_nothing_and_chat_leaves_voice_alone() {
let (mut c, f, t) = chat_conv(CallSettings::default());
let (mut c, f, t) = chat_conv(test_calls());
c.on_event(&fin(A, "자비스 날씨", 100), t);
c.on_event(&chat(CH_ALL, 9, B, false, "안녕"), t);
let did = f.take();
@@ -999,7 +1070,7 @@ mod tests {
#[test]
fn a_chat_answer_after_its_keep_time_is_not_posted() {
let (mut c, f, t) = chat_conv(CallSettings::default());
let (mut c, f, t) = chat_conv(test_calls());
c.on_event(&chat(CH_WAKE, 5, A, false, "자비스 안녕"), t);
let Did::ChatAnswer(id, ..) = f.take().remove(0) else { panic!() };
c.on_event(&CoreEvent::ReplyText { id: id.clone(), text: "안녕하세요.".into(), failed: false }, t + CHAT_ANSWER_KEPT);
@@ -1011,9 +1082,95 @@ mod tests {
let (mut c, f, t) = conv(PolicyKind::InOrder, CallMode::WakeWord);
c.on_event(&fin(A, "자비스 하나", 0), ms(t, 0));
c.on_event(&fin(B, "자비스 둘", 0), ms(t, 1000));
c.on_event(&fin(A, "자비스 셋", 900), ms(t, 1500)); // ended at 600, before 둘
c.on_event(&fin(C, "자비스 셋", 900), ms(t, 1500)); // ended at 600, before 둘
f.take();
c.on_event(&end("s1", false), ms(t, 2000));
assert_eq!(f.take(), vec![Did::Answer(A, "자비스 셋".into())]);
assert_eq!(f.take(), vec![Did::Answer(C, "자비스 셋".into())]);
}
fn alone(kind: PolicyKind) -> (Conversation<Fake>, Fake, Instant) {
let (c, f, t) = conv(kind, CallMode::Addressed);
*f.people.borrow_mut() = vec![A];
(c, f, t)
}
fn speak_start(id: &str) -> CoreEvent {
CoreEvent::SpeakStart(id.into())
}
#[test]
fn a_speaker_who_goes_on_before_the_answer_is_heard_gets_one_answer_for_all_of_it() {
// Live 2026-11-19: each piece of one thought was answered on its own, one after another.
for kind in [PolicyKind::Latest, PolicyKind::InOrder, PolicyKind::First, PolicyKind::Important] {
let (mut c, f, t) = alone(kind);
c.on_event(&fin(A, "지금 화면 봐봐", 100), ms(t, 0));
assert_eq!(f.take(), vec![Did::Answer(A, "지금 화면 봐봐".into())], "{kind:?}");
// The next piece starts 0.8 s after the first ended, while Jarvis is still thinking.
c.on_event(&fin(A, "디코 화면 공유 창에 가려져 있잖아", 100), ms(t, 1800));
assert_eq!(f.take(), vec![Did::Cancel("s1".into()),
Did::Answer(A, "지금 화면 봐봐 디코 화면 공유 창에 가려져 있잖아".into())], "{kind:?}");
c.on_event(&fin(A, "앞으로 오게 해 줘", 100), ms(t, 4000));
assert_eq!(f.take(), vec![Did::Cancel("s2".into()),
Did::Answer(A, "지금 화면 봐봐 디코 화면 공유 창에 가려져 있잖아 앞으로 오게 해 줘".into())],
"{kind:?}");
assert_eq!(c.missed().count(), 0, "a joined piece is not a missed request ({kind:?})");
// The cancelled speech's end does not end the joined answer.
c.on_event(&end("s1", true), ms(t, 4100));
assert!(f.take().is_empty(), "{kind:?}");
}
}
#[test]
fn what_follows_a_wake_word_call_is_joined_even_without_the_wake_word() {
let (mut c, f, t) = conv(PolicyKind::Latest, CallMode::WakeWord);
c.on_event(&fin(A, "자비스 디코 화면 공유 보여", 100), ms(t, 0));
c.on_event(&fin(A, "내 화면 말이야", 100), ms(t, 2000));
assert_eq!(f.take(), vec![Did::Answer(A, "자비스 디코 화면 공유 보여".into()), Did::Cancel("s1".into()),
Did::Answer(A, "자비스 디코 화면 공유 보여 내 화면 말이야".into())]);
// Someone else's words never join it.
c.on_event(&fin(B, "나도 안 보여", 100), ms(t, 2500));
assert!(f.take().is_empty());
}
#[test]
fn once_the_answer_is_heard_or_the_pause_is_long_the_next_words_are_a_new_request() {
let (mut c, f, t) = alone(PolicyKind::Latest);
c.on_event(&fin(A, "계산기 켜 줘", 100), ms(t, 0));
c.on_event(&speak_start("s1"), ms(t, 1500));
c.on_event(&fin(A, "고마워", 100), ms(t, 2500));
assert_eq!(f.take(), vec![Did::Answer(A, "계산기 켜 줘".into()), Did::Cancel("s1".into()),
Did::Answer(A, "고마워".into())]);
// Still thinking, but the speaker paused longer than call.continueMs.
let (mut c, f, t) = alone(PolicyKind::InOrder);
let pause = test_calls().continue_window.as_millis() as u64;
c.on_event(&fin(A, "계산기 켜 줘", 100), ms(t, 0));
// The next utterance (1 s long) starts `pause` + 100 ms after the first ended.
c.on_event(&fin(A, "메모장도", 100), ms(t, pause + 1100));
assert_eq!(f.take(), vec![Did::Answer(A, "계산기 켜 줘".into())], "queued as its own request");
}
#[test]
fn a_stop_said_while_the_answer_is_prepared_still_stops_it() {
let (mut c, f, t) = conv(PolicyKind::Latest, CallMode::WakeWord);
c.on_event(&fin(A, "자비스 날씨 알려 줘", 100), ms(t, 0));
c.on_event(&fin(A, "자비스 멈춰", 100), ms(t, 1500));
assert_eq!(f.take(), vec![Did::Answer(A, "자비스 날씨 알려 줘".into()), Did::StopAll, Did::Cancel("s1".into())]);
}
#[test]
fn going_on_lets_the_brain_get_ready_for_the_joined_request() {
let (mut c, f, t) = alone(PolicyKind::Latest);
c.on_event(&fin(A, "계산기 켜 줘", 100), ms(t, 0));
f.take();
c.on_event(&partial(A, "제일 위로"), ms(t, 1200));
assert_eq!(f.take(), vec![Did::Prepare(A, "계산기 켜 줘 제일 위로".into())]);
}
#[test]
fn the_joined_request_gets_no_second_earcon() {
let (mut c, f, t) = conv_earcon(PolicyKind::Latest);
c.on_event(&fin(A, "자비스 날씨", 100), ms(t, 0));
c.on_event(&fin(A, "부산", 100), ms(t, 1500));
assert_eq!(f.take(), vec![Did::Earcon, Did::Answer(A, "자비스 날씨".into()), Did::Cancel("s1".into()),
Did::Answer(A, "자비스 날씨 부산".into())]);
}
}

View File

@@ -24,7 +24,7 @@ Settings (`settings.json`, section `call`; wake words stay in `voice.wakeWords`)
| Key | Default | Meaning |
|---|---|---|
| `voice.wakeWords` | `["자비스"]` | Wake words for voice (also the STT hint). |
| `voice.wakeWords` | `["비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"]` | Wake words for voice (also the STT hint). The default calls Jarvis like a secretary, by the word and its usual call forms (user request, 2026-11-19; before it was `["자비스"]`). |
| `call.chatWakeWords` | `null` | Wake words for chat; `null` uses `voice.wakeWords`. |
| `call.similarity` | `0.2` | How far a heard word may be from a wake word, as a share of the wake word's normalised letters (0-0.5). `0` needs an exact match. |
| `call.mode` | `"wakeWord"` | `wakeWord`: answer only when called by a wake word. `addressed`: also answer what is judged to be said to Jarvis. `everything`: answer every utterance. |
@@ -34,6 +34,7 @@ Settings (`settings.json`, section `call`; wake words stay in `voice.wakeWords`)
| `call.stopWords` | `["멈춰", "그만", "정지", "stop"]` | Said together with a wake word, they stop Jarvis in any policy. |
| `call.earcon` | `true` | Play the earcon (below) when Jarvis takes a voice call. |
| `call.followUpMs` | `1500` | After a wake word said alone, how long the speaker's next utterance may take to start and still be joined to it. 0-5000. |
| `call.continueMs` | `5000` | Going on (below): how long after a request's utterance ended the speaker's next utterance may start and still be part of it, while its answer is not heard yet. 0 turns joining off. 0-15000. |
Normalisation: Unicode NFKD, lower case, and only letters and digits kept (spaces, punctuation and
combining marks dropped). NFKD splits Hangul syllables into their letters, so "자비즈" is one letter
@@ -42,6 +43,9 @@ away from "자비스" (of six) while "서비스" is two away.
A wake word is heard when some stretch of the normalised text is at most
`floor(similarity × letters of the wake word)` edits (insert, delete, substitute) from it. With the
default 0.2, "자비스" (six letters) allows one edit. A wake word anywhere in the utterance counts.
The earliest heard wake word is the call; where several are heard starting there, the one with the
fewest edits, then the longest, so a call form ("비서야") is removed whole and leaves nothing over
("비서야 멈춰" is a stop), and a near match of a longer word does not eat into what follows.
Decision for one utterance (`judge(speaker, text)`), in order:
@@ -75,6 +79,27 @@ connection (`WorkerLost`): the final it waited for will not come, and the restar
utterances are not continuations of it. Chat messages are not joined:
a message is complete as sent.
**Going on.** People say one thing in pieces, with pauses of a fraction of a second to several
seconds between them (live 2026-11-19: 0.1-4.7 s inside one thought), and every pause longer than
the worker's `end_silence_ms` ends an utterance. Each piece used to be answered on its own, one after
another, and the answers piled up (user request, 2026-11-19: what is said in pieces is one request).
So a speaker's voice request stays open until its answer is heard (the first audio of its speech
plays, `SpeakStart`), and while it is open their next final is joined to it when that utterance
started within `call.continueMs` of the request's end (start = the final's arrival minus its
`wait_ms` and `audio_ms`):
- The earlier request is taken back (the policy's `withdraw`): a running answer is stopped (its
speech `cancel`led, which also stops the brain, `voice_worker.spec.md`), a waiting one leaves the
queue, a pending "addressed" question is void. It is not a missed request.
- The joined text (the earlier text, a space, the new final) is judged and answered as one request,
as if it had been said in one go; it ends when the new final ended. It may be joined again.
- The new final is joined whatever it is on its own (a piece needs no wake word of its own), except
a stop: a stop stops, as always.
- Other speakers' utterances never join it. An answer already heard, an answer that ended, a missed
request, a stop, leaving and a lost worker end it; then what the speaker says is a new request.
- The joined request gets no second earcon when the first one's already played.
- While a request is open, its speaker's partials are sent to get ready as the joined text, also while
its answer is being written (it is the request that will be answered).
**Earcon.** A short two-tone "띠링" (about 0.3 s, made by the app, `app/src/earcon.rs`) tells the
speaker Jarvis heard the call, long before the answer is written: it is the first sound after a call.
It plays right before each voice answer starts (the policy's `Answer`), and at once when a wake word
@@ -113,6 +138,9 @@ Rules shared by every policy:
than the request's `ended_at` (in `important`: any ending utterance, since it competes on score), the
policy waits for that final, at most `settleMs` from the first waiting request's arrival. A final that
is not a call ends the wait for it. `latest` does not settle: an earlier utterance would lose anyway.
- **Taking back.** A request whose speaker went on (Going on) is withdrawn: stopped when it is the
running answer, removed when it waits, dropped from a pending ranking. Not missed; the joined
request that follows goes through the policy as any new request.
- **Stop.** A stop decision stops the current answer and drops every waiting request, in every policy.
The stopped answer and the dropped requests are missed (`stopped`).
- An answer that is already done is never started again.
@@ -178,7 +206,8 @@ event is passed on unchanged). It talks to the core through a small `Voice` inte
one request, stop all, earcon, prepare, ask "addressed", ask "rank"), so the glue is tested with a fake.
- `Chat` and `ReplyText`: chat calls, above.
- `Partial` (getting ready): a partial transcript that call detection already reads as a request
- `Partial` (getting ready): a partial of a speaker whose request is open (Going on) is sent as the
joined text. Otherwise a partial transcript that call detection already reads as a request
(joined to the speaker's waiting wake word alone, as its final would be; not the wake word alone,
not a stop) is sent to the brain as `prepare(user, text)`, so the brain judges what the call needs
while the speaker is still talking (`voice_worker.spec.md`, `prepare`). Not while an answer runs:
@@ -195,6 +224,7 @@ one request, stop all, earcon, prepare, ask "addressed", ask "rank"), so the glu
cannot be queued the answer is done at once); `StopAnswer` → `cancel` of that answer's speech;
`Rank { id }` → `rank` with id `k<id>` and the candidates' request ids; `Missed` → a debug log and
the missed list (the last 100: speaker, reason, time and length, never the text).
- `SpeakStart` of an answer's speech: the answer is heard, its request is no longer open (Going on).
- `SpeakEnd` of an answer's speech is that answer's done. The end of a speech the loop already
cancelled starts nothing.
- `Stopped` (all speech stopped from elsewhere: the slash command, `stop()`, the session ending) and
@@ -208,14 +238,17 @@ each to the chat model, the heard text fenced as data. Eval: `evals/test_voice_j
## Tests
`cargo test` covers call detection (exact and near wake words in Hangul and Latin, a different word
two letters away, wake word anywhere, stop with and without a wake word, designated combinations,
two letters away, wake word anywhere, the longest call form where several start (the default secretary call forms), stop with and without a wake word, designated combinations,
modes) and every policy rule above with fake times (settling, ordering by end time, busy, queue limit,
interruption and `response.interrupt` off, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a
fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed alone
with Jarvis answered without asking, the room counts sent with the question, addressed yes,
no and timeout, the earcon before each answer, once for a wake word alone, none while busy or when
off, partials that read as a call sent to get ready (joined to a waiting wake word, none while
answering), interruption cancelling the right speech, a ranking deciding, end times from
answering someone else, the joined text while going on), going on (pieces joined into one answer in
every policy without a missed record, a piece without a wake word joined, other speakers never, a
new request once the answer is heard or the pause is longer than `call.continueMs`, a stop still
stopping, no second earcon), interruption cancelling the right speech, a ranking deciding, end times from
`wait_ms`, and chat: an answer posted as a reply once, per-channel switches, no answers to bots or to
a stop, chat wake words and designated people, a failed answer posting nothing). `tests/worker_protocol.rs` checks an answer spoken under its id and both judgements against
the real Python worker with a fake brain.

View File

@@ -856,6 +856,68 @@ fn report_voice_state(inner: &Inner) {
debug_log!("voice", "reported voice {channel} with {n} members to the worker");
}
/// One `screen_action` against the real screen.
async fn screen_act(inner: &Arc<Inner>, req: crate::worker_client::ScreenAction) -> Result<serde_json::Value, String> {
let settings = inner.settings.clone();
tokio::task::spawn_blocking(move || match crate::screen::system_screen() {
Some(screen) => crate::screen::act(&settings, screen.as_ref(), &req),
None => Err("이 PC에서는 화면을 볼 수 없습니다".into()),
}).await.unwrap_or_else(|_| Err("화면을 캡처하지 못했습니다".into()))
}
/// How long a voice request waits for a screen share to be popped out, and how often it looks.
const SHARE_WAIT: Duration = Duration::from_secs(30);
const SHARE_POLL: Duration = Duration::from_secs(1);
/// Jarvis's voice channel's name and the people sharing their screen there (ID, display name);
/// None outside a voice channel.
fn share_state(inner: &Inner) -> Option<(Option<String>, Vec<(u64, String)>)> {
let (guild, channel) = inner.session.lock().unwrap().as_ref().map(|s| (s.guild_id, s.channel_id))?;
// IDs first, then names: the cache's guild is not held while `name_of` reads it again.
let (name, users): (Option<String>, Vec<u64>) = inner.cache.get()
.and_then(|c| c.guild(GuildId::new(guild)).map(|g| (
g.channels.get(&ChannelId::new(channel)).map(|c| c.name.clone()),
g.voice_states.values()
.filter(|v| v.channel_id.map(|c| c.get()) == Some(channel) && v.self_stream == Some(true))
.map(|v| v.user_id.get())
.collect())))
.unwrap_or_default();
let bot = inner.bot_id.get().copied();
Some((name, users.into_iter().filter(|u| Some(*u) != bot).map(|u| (u, name_of(inner, u))).collect()))
}
/// Someone's Discord screen share (admin.spec.md, "Screen viewing"): only while the person asked
/// about shares their screen in Jarvis's voice channel, from the window Discord pops it out to. With
/// no such window a voice request asks for it in the channel and waits `SHARE_WAIT`.
async fn share_capture(inner: &Arc<Inner>, mut req: crate::worker_client::ScreenAction) -> Result<serde_json::Value, String> {
// Who may see comes first: nothing is said or waited for for someone who may not.
if let Some(why) = crate::screen::refusal(&inner.settings, &req.user) {
return Err(why);
}
let (channel, streaming) = share_state(inner).ok_or("자비스가 음성 채널에 있을 때만 화면공유를 볼 수 있습니다")?;
let name = crate::screen::share_streamer(&streaming, req.person.as_deref())?;
req.voice_channel = channel;
let first = screen_act(inner, req.clone()).await;
if first.as_ref().err().map(String::as_str) != Some(crate::screen::NO_SHARE_WINDOW) {
return first;
}
let prompt = crate::screen::share_prompt(name.as_deref());
if req.via.as_deref() != Some("voice") {
return Err(format!("{} 그 뒤 다시 물어봐 주세요", prompt.trim_end_matches('.')));
}
debug_log!("conversation", "screen share {}: no popped-out window, asking and waiting", req.id);
with_playback(inner, |p| p.say(&prompt, false));
let until = tokio::time::Instant::now() + SHARE_WAIT;
while tokio::time::Instant::now() < until {
tokio::time::sleep(SHARE_POLL).await;
let r = screen_act(inner, req.clone()).await;
if r.as_ref().err().map(String::as_str) != Some(crate::screen::NO_SHARE_WINDOW) {
return r;
}
}
Err("화면공유 창이 분리되지 않아 볼 수 없습니다".into())
}
/// The registered chat channels and their switches, for the brain's chat tool.
fn report_chat_channels(inner: &Inner) {
inner.worker.chat_channels(inner.bot_id.get().copied(), &inner.settings.chat_channels);
@@ -1003,12 +1065,11 @@ async fn worker_task(core: std::sync::Weak<Inner>, mut rx: mpsc::UnboundedReceiv
let inner = inner.clone();
tokio::spawn(async move {
let started = std::time::Instant::now();
let settings = inner.settings.clone();
let req = a.clone();
let result = tokio::task::spawn_blocking(move || match crate::screen::system_screen() {
Some(screen) => crate::screen::act(&settings, screen.as_ref(), &req),
None => Err("이 PC에서는 화면을 볼 수 없습니다".into()),
}).await.unwrap_or_else(|_| Err("화면을 캡처하지 못했습니다".into()));
let result = if a.action == "capture" && a.target.as_deref() == Some("share") {
share_capture(&inner, a.clone()).await
} else {
screen_act(&inner, a.clone()).await
};
// The kind of target, size and time only: never a title or a picture.
let target = crate::screen::target_of(&a).map(|(t, _)| crate::screen::activity_target(&t))
.unwrap_or_else(|_| "?".into());

View File

@@ -719,7 +719,8 @@ impl Pc {
if self.stop.on() {
return Err(refused(STOPPED));
}
if w.minimised {
// Keys restore in their moment in front (restoring brings the window forward).
if w.minimised && a.action != "keys" {
self.cmd(&args(&["restore", "--window-id", &w.id]), None, &a.action)?;
}
match a.action.as_str() {
@@ -743,15 +744,16 @@ impl Pc {
}
/// Press `keys` in window `w` one way, chosen first (admin.spec.md, "Keys"): behind it when it is
/// one plain key and the window takes it so, else in front for a moment and the window that was
/// in front brought back. Never both ways: a key is never pressed twice.
/// one plain key and the window takes it so, else in front for a moment (a minimised window
/// restored in it) and the window that was in front brought back. Never both ways: a key is
/// never pressed twice.
fn keys(&self, w: &Win, keys: &str) -> Result<(), Fail> {
let front = self.desktop.foreground();
if front.as_deref() == Some(w.id.as_str()) {
crate::debug_log!("conversation", "pc keys way=front-already");
return self.press(w, keys);
}
if behind_key(keys).is_some() && self.desktop.takes_keys_behind(&w.id) {
if !w.minimised && behind_key(keys).is_some() && self.desktop.takes_keys_behind(&w.id) {
crate::debug_log!("conversation", "pc keys way=behind");
return self.desktop.post_key(&w.id, keys).map_err(|code| failed(&code, "keys"));
}
@@ -760,7 +762,9 @@ impl Pc {
return Err(refused(NOT_IN_FRONT_NOW));
}
crate::debug_log!("conversation", "pc keys way=front");
let pressed = self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys")
let restored = if w.minimised { self.cmd(&args(&["restore", "--window-id", &w.id]), None, "keys").map(|_| ()) } else { Ok(()) };
let pressed = restored
.and_then(|_| self.cmd(&args(&["focus-window", "--window-id", &w.id]), None, "keys"))
.and_then(|_| self.press(w, keys));
if let Some(back) = front {
if self.desktop.run(&args(&["focus-window", "--window-id", &back]), None, self.stop).is_err() {
@@ -1361,7 +1365,8 @@ mod tests {
}
"get" => json!({"property": "states", "value": if self.secure { json!(["secure"]) } else { json!([]) }}),
"launch" => self.launched.clone(),
"focus-window" => {
// restore activates the window (SW_RESTORE), as focus-window does.
"focus-window" | "restore" => {
*self.front.lock().unwrap() = Some(args[2].clone());
json!({"disposition": {"delivery": "delivered_unverified"}})
}
@@ -1743,6 +1748,26 @@ mod tests {
assert_eq!(fake.front().as_deref(), Some("w-2"));
}
#[test]
fn keys_to_a_minimised_window_bring_back_the_window_that_was_in_front_and_wait_for_no_game() {
// Restoring brings the window forward, so it is part of the moment in front.
let (pc, fake, _) = keys_auto(Fake { behind: true, ..Fake::new() });
assert_eq!(pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "ctrl+a"}))).reply["ok"], true);
assert_eq!(fake.calls_from("restore"), [
args(&["restore", "--window-id", "w-3"]),
args(&["focus-window", "--window-id", "w-3"]),
args(&["press", "ctrl+a", "--window-id", "w-3"]),
args(&["focus-window", "--window-id", "w-2"]),
args(&["snapshot", "--window-id", "w-3", "--compact", "--timeout-ms", SNAPSHOT_MS])]);
assert_eq!(fake.front().as_deref(), Some("w-2"), "the window in front is back");
// During a game or full screen a minimised window is not even restored.
let (pc, fake, _) = keys_auto(Fake { behind: true, fullscreen: true, ..Fake::new() });
let o = pc.act(&action("keys", OWNER, json!({"window": "calc", "keys": "a"})));
assert_eq!(o.reply, json!({"ok": false, "error": NOT_IN_FRONT_NOW}));
assert!(!fake.commands().contains(&"restore".to_owned()), "{:?}", fake.commands());
assert_eq!(fake.front().as_deref(), Some("w-2"));
}
#[test]
fn a_store_app_agent_desktop_cannot_name_gets_its_keys_once_it_is_in_front() {
// Brought in front, and already in front: the key is pressed once, in that window.

View File

@@ -254,6 +254,21 @@ impl Policy {
}
}
/// The request is taken back before its answer was heard: its speaker went on talking and it is
/// joined to what they said next (conversation.spec.md, "Going on"). Not a missed request. When
/// it is the running answer it is stopped; nothing else starts until the joined request comes.
pub fn withdraw(&mut self, id: u64) -> Vec<Action> {
if self.current.as_ref().is_some_and(|r| r.id == id) {
self.current = None;
return vec![Action::StopAnswer(id)];
}
self.waiting.retain(|r| r.id != id);
if self.ranking.as_ref().is_some_and(|r| r.ids.contains(&id)) {
self.ranking = None;
}
vec![]
}
/// A stop command: stop the answer and drop everything waiting.
pub fn stop(&mut self) -> Vec<Action> {
let mut out = vec![];

View File

@@ -96,6 +96,8 @@ pub enum Target {
Screen,
Monitor(u32),
Window(String),
/// Someone's Discord screen share, in the window Discord pops it out to.
Share,
}
/// The activity log's target: never a title.
@@ -104,6 +106,7 @@ pub fn activity_target(t: &Target) -> String {
Target::Screen => "전체 화면".into(),
Target::Monitor(n) => format!("모니터 {n}"),
Target::Window(_) => "창".into(),
Target::Share => "화면공유".into(),
}
}
@@ -120,6 +123,7 @@ pub fn target_of(a: &ScreenAction) -> Result<(Target, Detail), String> {
let w = a.window.as_deref().map(str::trim).filter(|w| !w.is_empty()).ok_or("창 이름이 필요합니다")?;
Target::Window(w.chars().take(200).collect())
}
Some("share") => Target::Share,
_ => Target::Screen,
};
Ok((target, detail))
@@ -169,6 +173,55 @@ pub fn pick_window<'a>(s: &'a Settings, all: &'a [Window], text: &str) -> Option
}
/// Whether nothing shows (protected video, exclusive full screen): every pixel near black.
/// Discord's programs (stable, PTB, Canary).
const DISCORD: [&str; 3] = ["discord.exe", "discordptb.exe", "discordcanary.exe"];
/// What a share capture gives while Discord has no popped-out screen share window: the core then
/// asks for one and waits (admin.spec.md, "Screen viewing").
pub const NO_SHARE_WINDOW: &str = "분리된 화면공유 창이 없습니다";
/// The window Discord pops a screen share out to: a Discord window that is not its main one (whose
/// title ends with "Discord"). Its title is the voice channel's name, so the one naming Jarvis's
/// voice channel goes first, else the one nearest the front.
pub fn share_window<'a>(s: &'a Settings, all: &'a [Window], channel: Option<&str>) -> Option<&'a Window> {
let shares: Vec<&Window> = listable(s, all)
.filter(|w| DISCORD.contains(&w.program.as_str()) && !w.title.trim().to_lowercase().ends_with("discord"))
.collect();
let channel = channel.map(|c| c.trim().to_lowercase()).filter(|c| !c.is_empty());
channel.and_then(|c| shares.iter().find(|w| w.title.to_lowercase().contains(&c)).copied())
.or_else(|| shares.first().copied())
}
/// Letters and digits in lower case, so a spoken name and a display name compare.
fn name_key(name: &str) -> String {
name.chars().filter(|c| c.is_alphanumeric()).flat_map(char::to_lowercase).collect()
}
/// Whose share the popped-out window should show, from the people sharing their screen in Jarvis's
/// voice channel (ID, display name): the person asked about (their name inside what was said, or
/// what was said inside their name), or with no name anyone sharing (`None` when several do).
pub fn share_streamer(streaming: &[(u64, String)], person: Option<&str>) -> Result<Option<String>, String> {
let person = person.map(str::trim).filter(|p| !name_key(p).is_empty());
let Some(person) = person else {
return match streaming {
[] => Err("지금 음성 채널에서 화면공유 중인 사람이 없습니다".into()),
[(_, name)] => Ok(Some(name.clone())),
_ => Ok(None),
};
};
let key = name_key(person);
streaming.iter()
.find(|(_, name)| { let n = name_key(name); !n.is_empty() && (key.contains(&n) || n.contains(&key)) })
.map(|(_, name)| Some(name.clone()))
.ok_or_else(|| format!("{}님은 지금 화면공유 중이 아닙니다", person.chars().take(40).collect::<String>()))
}
/// What Jarvis says in the voice channel while it waits for the share to be popped out.
pub fn share_prompt(name: Option<&str>) -> String {
let whose = name.map(|n| format!("{n}님 화면공유를")).unwrap_or_else(|| "화면공유를".into());
format!("디스코드에서 {whose} 시청하고 창 분리 버튼을 눌러 주세요.")
}
pub fn all_black(img: &Image) -> bool {
img.bgra.chunks_exact(4).all(|p| p[0] <= 8 && p[1] <= 8 && p[2] <= 8)
}
@@ -330,6 +383,16 @@ pub fn act(s: &Settings, screen: &dyn Screen, a: &ScreenAction) -> Result<Value,
let black = all_black(&img);
(img, black, format!("창 {}", w.program.strip_suffix(".exe").unwrap_or(&w.program)))
}
Target::Share => {
let all = screen.windows()?;
let w = share_window(s, &all, a.voice_channel.as_deref()).ok_or(NO_SHARE_WINDOW)?;
if w.minimised {
return Err("화면공유 창이 최소화되어 있어 볼 수 없습니다. 창을 다시 열어 주세요".into());
}
let img = screen.capture_window(w)?;
let black = all_black(&img);
(img, black, activity_target(&target))
}
_ => {
let ms = screen.monitors()?;
let chosen: Vec<Monitor> = match target {
@@ -813,6 +876,73 @@ mod tests {
assert!(v.get("images").is_none());
}
fn with_share(f: &mut Fake, popout: Window) {
f.windows.insert(1, win("discord.exe", "#일반 | 우리 서버 - Discord", Rect { x: 0, y: 0, w: 1200, h: 800 }));
f.windows.insert(0, popout);
}
#[test]
fn a_screen_share_is_seen_in_discords_popped_out_window_never_its_main_one() {
let s = settings();
let mut f = fake();
let mut a = action("capture", OWNER, json!({"target": "share", "person": "철수"}));
a.voice_channel = Some("게임방".into());
// Only the main window: nothing popped out yet.
f.windows.insert(1, win("discord.exe", "#일반 | 우리 서버 - Discord", Rect { x: 0, y: 0, w: 1200, h: 800 }));
assert_eq!(act(&s, &f, &a).unwrap_err(), NO_SHARE_WINDOW);
assert!(f.captured.borrow().is_empty());
let mut f = fake();
with_share(&mut f, win("discord.exe", "게임방", Rect { x: 0, y: 0, w: 960, h: 540 }));
let v = act(&s, &f, &a).unwrap();
assert_eq!(*f.captured.borrow(), vec!["window discord.exe"]);
assert_eq!(v["item"]["what"], "화면공유");
assert_eq!(activity_target(&target_of(&a).unwrap().0), "화면공유");
// A popped-out window that is minimised is asked to be opened again.
let mut f = fake();
with_share(&mut f, Window { minimised: true, ..win("discord.exe", "게임방", Rect { x: 0, y: 0, w: 960, h: 540 }) });
assert!(act(&s, &f, &a).unwrap_err().contains("최소화"));
// Discord on the exclusion list is never seen, popped out or not.
let mut s = settings();
s.pc_control.screen_exclude.push("Discord".into());
let mut f = fake();
with_share(&mut f, win("discord.exe", "게임방", Rect { x: 0, y: 0, w: 960, h: 540 }));
assert_eq!(act(&s, &f, &a).unwrap_err(), NO_SHARE_WINDOW);
}
#[test]
fn the_popped_out_window_naming_jarvis_voice_channel_goes_first() {
let s = settings();
let all = vec![
win("discordptb.exe", "다른 방", Rect { x: 0, y: 0, w: 10, h: 10 }),
win("discordptb.exe", "게임방", Rect { x: 0, y: 0, w: 10, h: 10 }),
win("discordptb.exe", "Discord", Rect { x: 0, y: 0, w: 10, h: 10 }),
];
assert_eq!(share_window(&s, &all, Some("게임방")).unwrap().title, "게임방");
assert_eq!(share_window(&s, &all, None).unwrap().title, "다른 방", "else the one nearest the front");
assert!(share_window(&s, &all[2..], Some("게임방")).is_none(), "the main window is never the share");
}
#[test]
fn a_share_is_seen_only_while_the_person_asked_about_shares_their_screen() {
let streaming = vec![(1, "철수".to_owned()), (2, "Younghee".to_owned())];
assert_eq!(share_streamer(&streaming, Some("철수님")), Ok(Some("철수".to_owned())), "a spoken suffix still matches");
assert_eq!(share_streamer(&streaming, Some("younghee")), Ok(Some("Younghee".to_owned())));
assert_eq!(share_streamer(&streaming, Some("민수")).unwrap_err(), "민수님은 지금 화면공유 중이 아닙니다");
assert_eq!(share_streamer(&streaming, None), Ok(None), "no name: whatever the window shows");
assert_eq!(share_streamer(&streaming[..1], Some(" ")), Ok(Some("철수".to_owned())));
assert!(share_streamer(&[], None).unwrap_err().contains("화면공유 중인 사람이 없습니다"));
assert!(share_streamer(&[], Some("철수")).unwrap_err().contains("철수님은"));
}
#[test]
fn the_share_prompt_names_the_person_when_it_is_known() {
assert!(share_prompt(Some("철수")).contains("철수님 화면공유"));
assert!(share_prompt(None).contains("화면공유를"));
}
#[test]
fn scaling_averages_the_area() {
let mut img = solid(4, 2, 0);

View File

@@ -254,6 +254,12 @@ impl std::error::Error for SettingsError {}
pub const DEFAULT_JOIN_NOTICE: &str =
"자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다.";
/// The default wake words: Jarvis is called like a secretary, by the word and its usual call forms
/// (the longest one heard is the call, `calls.rs`).
pub fn default_wake_words() -> Vec<String> {
["비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"].map(String::from).to_vec()
}
impl Default for Settings {
fn default() -> Self {
Self {
@@ -268,7 +274,7 @@ impl Default for Settings {
listen_to_bots: false,
record: false,
join_notice: DEFAULT_JOIN_NOTICE.into(),
wake_words: vec!["자비스".into()],
wake_words: default_wake_words(),
stt: Map::new(),
tts_speed: 1.5,
},
@@ -447,6 +453,7 @@ fn parse_call(raw: &Map<String, Value>, wake_words: &[String], d: &CallSettings)
designated_no_wake_word: get_bool(call, "designatedNoWakeWord", "call", d.designated_no_wake_word)?,
stop_words,
follow_up: get_ms(call, "followUpMs", "call", d.follow_up, 5000)?,
continue_window: get_ms(call, "continueMs", "call", d.continue_window, 15000)?,
earcon: get_bool(call, "earcon", "call", d.earcon)?,
},
chat_wake_words,

View File

@@ -15,10 +15,13 @@ use crate::settings::{parse_settings, DEFAULT_JOIN_NOTICE};
#[serde(rename_all = "camelCase")]
pub enum Screen {
Bot,
Channels,
VoiceChannel,
Call,
Response,
Chat,
Memory,
Model,
Voice,
Llm,
Features,
Collect,
Game,
@@ -78,33 +81,34 @@ pub fn fields() -> Vec<Field> {
vec![
f("discord.commandName", Bot, Text { max: 32, non_empty: true }, json!("자비스"), "슬래시 명령 이름"),
f("discord.ephemeralReplies", Bot, Bool, json!(true), "명령 응답은 부른 사람만 보기"),
f("discord.guildId", Channels, Id, Value::Null, "서버"),
f("discord.voiceChannelId", Channels, Id, Value::Null, "자동으로 들어갈 음성 채널"),
f("voice.maxSpeakers", Channels, Int { min: 1, max: 10 }, json!(10), "동시에 받아 적을 최대 인원"),
f("voice.listenToBots", Channels, Bool, json!(false), "다른 봇의 말도 받아 적기"),
f("voice.wakeWords", Channels, TextList { max_items: 20, nullable: false }, json!(["자비스"]), "호출어"),
f("call.mode", Channels, Choice { values: &["wakeWord", "addressed", "everything"] }, json!("wakeWord"),
f("discord.guildId", VoiceChannel, Id, Value::Null, "서버"),
f("discord.voiceChannelId", VoiceChannel, Id, Value::Null, "자동으로 들어갈 음성 채널"),
f("voice.maxSpeakers", VoiceChannel, Int { min: 1, max: 10 }, json!(10), "동시에 받아 적을 최대 인원"),
f("voice.listenToBots", VoiceChannel, Bool, json!(false), "다른 봇의 말도 받아 적기"),
f("voice.wakeWords", Call, TextList { max_items: 20, nullable: false }, json!(crate::settings::default_wake_words()), "호출어"),
f("call.mode", Call, Choice { values: &["wakeWord", "addressed", "everything"] }, json!("wakeWord"),
"호출 방식"),
f("call.similarity", Channels, Number { min: 0.0, max: 0.5 }, json!(0.2), "비슷한 발음 허용 정도"),
f("call.designated", Channels, IdList, json!([]), "지정한 사람"),
f("call.designatedOnly", Channels, Bool, json!(false), "지정한 사람에게만 응답"),
f("call.designatedNoWakeWord", Channels, Bool, json!(false), "지정한 사람은 호출어 없이 대답"),
f("call.stopWords", Channels, TextList { max_items: 20, nullable: false },
f("call.similarity", Call, Number { min: 0.0, max: 0.5 }, json!(0.2), "비슷한 발음 허용 정도"),
f("call.designated", Call, IdList, json!([]), "지정한 사람"),
f("call.designatedOnly", Call, Bool, json!(false), "지정한 사람에게만 응답"),
f("call.designatedNoWakeWord", Call, Bool, json!(false), "지정한 사람은 호출어 없이 대답"),
f("call.stopWords", Call, TextList { max_items: 20, nullable: false },
json!(["멈춰", "그만", "정지", "stop"]), "정지 명령"),
f("call.followUpMs", Channels, Int { min: 0, max: 5000 }, json!(1500), "호출어 뒤 이어 말하기 대기(ms)"),
f("call.earcon", Channels, Bool, json!(true), "부르면 효과음"),
f("response.policy", Channels, Choice { values: &["first", "inOrder", "latest", "important"] },
f("call.followUpMs", Call, Int { min: 0, max: 5000 }, json!(1500), "호출어 뒤 이어 말하기 대기(ms)"),
f("call.continueMs", Call, Int { min: 0, max: 15000 }, json!(5000), "끊어 말한 문장 합치기(ms)"),
f("call.earcon", Call, Bool, json!(true), "부르면 효과음"),
f("response.policy", Response, Choice { values: &["first", "inOrder", "latest", "important"] },
json!("latest"), "동시 대화 응답 정책"),
f("response.queueMax", Channels, Int { min: 1, max: 20 }, json!(5), "기다리는 요청 최대 개수"),
f("response.collectMs", Channels, Int { min: 0, max: 3000 }, json!(700), "중요도 비교 대기(ms)"),
f("response.settleMs", Channels, Int { min: 0, max: 3000 }, json!(800), "먼저 끝난 말 전사 대기(ms)"),
f("response.priorities", Channels, IdNumberMap, json!({}), "사람별 우선순위"),
f("response.interrupt", Channels, Bool, json!(true),
f("response.queueMax", Response, Int { min: 1, max: 20 }, json!(5), "기다리는 요청 최대 개수"),
f("response.collectMs", Response, Int { min: 0, max: 3000 }, json!(700), "중요도 비교 대기(ms)"),
f("response.settleMs", Response, Int { min: 0, max: 3000 }, json!(800), "먼저 끝난 말 전사 대기(ms)"),
f("response.priorities", Response, IdNumberMap, json!({}), "사람별 우선순위"),
f("response.interrupt", Response, Bool, json!(true),
"말하는 중에 새 요청이 오면 하던 말 멈추기(가장 최근 것, 중요한 것 먼저)"),
f("voice.joinNotice", Channels, Text { max: 2000, non_empty: true }, json!(DEFAULT_JOIN_NOTICE),
f("voice.joinNotice", VoiceChannel, Text { max: 2000, non_empty: true }, json!(DEFAULT_JOIN_NOTICE),
"입장 시 전사 고지 문구"),
f("chatChannels", Chat, ChatChannels, json!([]), "채팅 채널"),
f("call.chatWakeWords", Chat, TextList { max_items: 20, nullable: true }, Value::Null,
f("call.chatWakeWords", Call, TextList { max_items: 20, nullable: true }, Value::Null,
"채팅 호출어(비우면 음성과 같음)"),
// The worker's memory settings and their defaults (voice_worker.spec.md, "Memory settings").
f("memory.chat", Memory, Bool, json!(true), "등록한 채팅 채널 참고"),
@@ -116,21 +120,21 @@ pub fn fields() -> Vec<Field> {
f("memory.recent_records", Memory, Int { min: 0, max: 200 }, json!(20), "최근 대화 원문 개수"),
f("memory.matches", Memory, Int { min: 0, max: 20 }, json!(5), "과거 기록 가져올 개수"),
f("memory.budget_tokens", Memory, Number { min: 0.0, max: 100000.0 }, json!(4000), "참고 분량 상한(토큰)"),
f("voice.record", Memory, Bool, json!(false),
f("voice.record", VoiceChannel, Bool, json!(false),
"목소리 녹음 저장(사람 말과 자비스 말, 분석용, 7일 보관, 다음 시작부터)"),
// The worker's STT settings and their defaults (voice_worker.spec.md, "STT").
f("voice.stt.end_silence_ms", Model, Int { min: 100, max: 2000 }, json!(300), "말 끝 판정 시간(ms)"),
f("voice.stt.partials", Model, Bool, json!(true), "중간 전사"),
f("voice.stt.partial_interval_ms", Model, Int { min: 100, max: 5000 }, json!(400), "중간 전사 간격(ms)"),
f("voice.stt.final_beam", Model, Int { min: 1, max: 5 }, json!(2), "최종 전사 정확도(빔)"),
f("voice.stt.language", Model, LanguageCode, json!("ko"), "받아 적을 언어(비우면 자동)"),
f("voice.stt.end_silence_ms", Voice, Int { min: 100, max: 2000 }, json!(300), "말 끝 판정 시간(ms)"),
f("voice.stt.partials", Voice, Bool, json!(true), "중간 전사"),
f("voice.stt.partial_interval_ms", Voice, Int { min: 100, max: 5000 }, json!(400), "중간 전사 간격(ms)"),
f("voice.stt.final_beam", Voice, Int { min: 1, max: 5 }, json!(2), "최종 전사 정확도(빔)"),
f("voice.stt.language", Voice, LanguageCode, json!("ko"), "받아 적을 언어(비우면 자동)"),
// MeloTTS's Korean model has one voice (speaker "KR"), so only the speed is a choice.
f("voice.tts.speed", Model, Number { min: 0.5, max: 2.0 }, json!(1.5), "말하기 속도(1.0 보통, 기본 1.5, 저장하면 바로 적용)"),
f("voice.tts.speed", Voice, Number { min: 0.5, max: 2.0 }, json!(1.5), "말하기 속도(1.0 보통, 기본 1.5, 저장하면 바로 적용)"),
// The LLM (admin.spec.md, "LLM"); `llm.url` (a server the app does not start) is file-only.
f("llm.context", Model, Int { min: 4096, max: 262144 }, json!(32768), "대화 모델이 한 번에 기억하는 양(토큰)"),
f("brain.location", Model, Text { max: 100, non_empty: false }, json!(crate::settings::DEFAULT_LOCATION),
f("llm.context", Llm, Int { min: 4096, max: 262144 }, json!(32768), "대화 모델이 한 번에 기억하는 양(토큰)"),
f("brain.location", Features, Text { max: 100, non_empty: false }, json!(crate::settings::DEFAULT_LOCATION),
"기본 위치(날씨, 검색, 추천의 기준, 비우면 자동 감지)"),
f("llm.port", Model, Int { min: 1024, max: 65535 }, json!(8080), "대화 모델 포트"),
f("llm.port", Llm, Int { min: 1024, max: 65535 }, json!(8080), "대화 모델 포트"),
// The brain's features (admin.spec.md, "Features and MCP servers").
f("features.weather", Features, Bool, json!(true), "날씨"),
f("features.webSearch", Features, Bool, json!(true), "웹 검색"),

View File

@@ -172,16 +172,25 @@ pub struct ScreenAction {
/// `capture` or `list`.
pub action: String,
pub user: String,
/// capture: `screen`, `monitor` or `window`.
/// `voice` or `chat`.
#[serde(default)]
pub via: Option<String>,
/// capture: `screen`, `monitor`, `window` or `share`.
#[serde(default)]
pub target: Option<String>,
#[serde(default)]
pub monitor: Option<u32>,
#[serde(default)]
pub window: Option<String>,
/// share: the name of the person whose screen share is asked about.
#[serde(default)]
pub person: Option<String>,
/// `fast`, `normal` or `detailed`.
#[serde(default)]
pub detail: Option<String>,
/// share: the name of Jarvis's voice channel, set by the core (the popped-out window's title).
#[serde(skip)]
pub voice_channel: Option<String>,
}
/// The brain's pcControl tool asks the app for the person asking (voice_worker.spec.md, "PC control").

View File

@@ -210,8 +210,8 @@ async fn conversation_round_trip_on_real_discord() {
}
std::fs::write(data.join("config").join("settings.json"), serde_json::to_string_pretty(&json!({
"discord": {"guildId": guild.to_string(), "voiceChannelId": voice.to_string()},
// The "person" here is a bot, so bots must be heard.
"voice": {"listenToBots": true},
// The "person" here is a bot, so bots must be heard; the recorded questions call "자비스".
"voice": {"listenToBots": true, "wakeWords": ["자비스"]},
"voiceWorker": {"python": env("JARVIS_VOICE_PYTHON"), "cwd": root.join("../src"), "args": args},
"llm": {"url": env("JARVIS_LIVE_LLM_URL")},
})).unwrap()).unwrap();

View File

@@ -52,6 +52,16 @@ impl Drop for Closed {
}
}
/// Minimise window `id` ("w-<handle>") as the person would.
#[cfg(windows)]
fn minimise(id: &str) {
let h: usize = id.trim_start_matches("w-").parse().unwrap();
// SAFETY: a plain window call on a handle from list-windows.
unsafe { windows_sys::Win32::UI::WindowsAndMessaging::ShowWindow(h as _, windows_sys::Win32::UI::WindowsAndMessaging::SW_MINIMIZE) };
}
#[cfg(not(windows))]
fn minimise(_id: &str) {}
fn wait_window(pc: &Pc, f: impl Fn(&Value) -> bool) -> String {
let until = Instant::now() + Duration::from_secs(15);
loop {
@@ -117,7 +127,20 @@ fn keys_go_behind_or_in_front_for_a_moment_and_the_window_in_front_stays() {
assert_eq!(r["ok"], true, "x: {r}");
let after = values(&act(&pc, "read", json!({"window": notepad})));
println!("notepad after ctrl+a, x: {after:?}");
assert!(after.contains('x') && !after.contains("hi"), "ctrl+a selected the text and x replaced it");
assert!(after.split(" | ").any(|v| v == "x") && !after.contains("hi"), "ctrl+a selected the text and x replaced it");
// A minimised window: restored in the moment in front, and the window in front is back.
minimise(&notepad);
std::thread::sleep(Duration::from_millis(500));
let _ = probe.run(&["focus-window".into(), "--window-id".into(), front.clone()], None, stop);
let r = act(&pc, "keys", json!({"window": notepad, "keys": "ctrl+a"}));
assert_eq!(r["ok"], true, "minimised ctrl+a: {r}");
assert_eq!(probe.foreground().as_deref(), Some(front.as_str()), "minimised ctrl+a: the window in front is back");
let r = act(&pc, "keys", json!({"window": notepad, "keys": "y"}));
assert_eq!(r["ok"], true, "y: {r}");
let after = values(&act(&pc, "read", json!({"window": notepad})));
println!("notepad after minimised, ctrl+a, y: {after:?}");
assert!(after.split(" | ").any(|v| v == "y") && !after.split(" | ").any(|v| v == "x"), "ctrl+a reached the restored Notepad");
// The Calculator: in front for a moment when it does not take keys behind (the Store app).
let calc = act(&pc, "run", json!({"program": "calc"}));

View File

@@ -992,7 +992,7 @@ function updateCard() {
async function checkUpdate() {
try {
const u = await api("update_check");
if (u.latest) { $("#nav-app").textContent = "앱 (새 버전)"; toast(`새 버전 ${u.latest.version}이 있습니다. 앱 화면에서 설치할 수 있습니다.`); }
if (u.latest) { $("#nav-app").textContent = "앱 (새 버전)"; setGroup($("#nav-app").closest(".group"), true); toast(`새 버전 ${u.latest.version}이 있습니다. 앱 화면에서 설치할 수 있습니다.`); }
} catch (e) { /* offline: the 앱 screen shows why */ }
}
@@ -1025,14 +1025,18 @@ async function show(name) {
current = name;
clearInterval(ticker);
for (const b of document.querySelectorAll(".nav")) b.classList.toggle("on", b.dataset.screen === name);
// The group of the screen shown opens (a new version's 앱 button is then seen too).
const group = document.querySelector(`.nav[data-screen="${name}"]`)?.closest(".group");
if (group) setGroup(group, true);
const screen = $("#screen");
try {
if (name === "dashboard") { screen.replaceChildren(); await refresh(); ticker = setInterval(refresh, 1000); return; }
if (name === "logs") { const l = logsScreen(); screen.replaceChildren(l.view); ticker = setInterval(l.tick, 1000); return; }
if (name === "conversations") { const c = conversationScreen(); screen.replaceChildren(c.view); ticker = setInterval(c.tick, 500); return; }
if (name === "notes") { screen.replaceChildren(notesScreen()); return; }
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능",
collect: "백그라운드 수집", game: "게임 모드", schedule: "예약", pcControl: "PC 제어", app: "앱" };
const titles = { bot: "봇", voiceChannel: "음성 채널", call: "호출", response: "여럿이 말할 때", chat: "채팅 채널",
memory: "대화 기억", voice: "받아 적기·말하기", llm: "대화 모델", features: "기능 켜기·MCP",
collect: "백그라운드 수집", game: "게임 모드", schedule: "예약", pcControl: "PC 제어·화면 보기", app: "앱" };
const update = name === "app" ? updateCard() : null;
const extra = name === "bot" ? await tokenCard() : name === "features" ? h("div", {}, await kmaKeyCard(), await displayDriverCard())
: name === "app" ? h("div", {}, update.view, h("div", { class: "hint", style: "margin:12px 0" }, "트레이로 최소화하면 창만 숨고 봇과 서비스는 계속 돕니다. 작업 표시줄 오른쪽의 Jarvis 아이콘을 누르면 창이 다시 열리고, 아이콘의 오른쪽 클릭 메뉴에서 완전 종료할 수 있습니다. 저장하면 바로 적용됩니다.")) : null;
@@ -1046,6 +1050,23 @@ async function show(name) {
} catch (e) { fail(e); }
}
/** Open or close one menu group; which groups are open is kept for the next start. */
function setGroup(g, open) {
g.classList.toggle("open", open);
g.querySelector(".group-head").setAttribute("aria-expanded", String(open));
const kept = new Set(JSON.parse(localStorage.getItem("openGroups") || "[]"));
open ? kept.add(g.dataset.group) : kept.delete(g.dataset.group);
localStorage.setItem("openGroups", JSON.stringify([...kept]));
}
function initGroups() {
const kept = new Set(JSON.parse(localStorage.getItem("openGroups") || "[]"));
for (const g of document.querySelectorAll(".group")) {
if (kept.has(g.dataset.group)) setGroup(g, true);
g.querySelector(".group-head").onclick = () => setGroup(g, !g.classList.contains("open"));
}
}
// ---- first-run wizard ---------------------------------------------------------------------------
const WIZ = ["데이터 폴더", "PC 점검", "모델", "디스코드 봇", "서버·채널", "호출어", "완료"];
@@ -1213,6 +1234,7 @@ async function start() {
$("#wizard").hidden = true;
$("#main").hidden = false;
for (const b of document.querySelectorAll(".nav")) b.onclick = () => show(b.dataset.screen);
initGroups();
show("dashboard");
checkUpdate();
}

View File

@@ -13,19 +13,52 @@
<div class="brand">Jarvis</div>
<button data-screen="dashboard" class="nav">대시보드</button>
<button data-screen="conversations" class="nav">대화</button>
<button data-screen="bot" class="nav">봇</button>
<button data-screen="channels" class="nav">채널·호출</button>
<button data-screen="chat" class="nav">채팅 채널</button>
<button data-screen="memory" class="nav">대화 기억</button>
<button data-screen="model" class="nav">모델</button>
<button data-screen="features" class="nav">기능</button>
<button data-screen="collect" class="nav">백그라운드 수집</button>
<button data-screen="schedule" class="nav">예약</button>
<button data-screen="pcControl" class="nav">PC 제어</button>
<button data-screen="notes" class="nav">사람별 지침</button>
<button data-screen="game" class="nav">게임 모드</button>
<button data-screen="logs" class="nav">로그</button>
<button data-screen="app" class="nav" id="nav-app">앱</button>
<div class="group" data-group="discord">
<button class="group-head" aria-expanded="false">디스코드</button>
<div class="group-items">
<button data-screen="bot" class="nav">봇</button>
<button data-screen="voiceChannel" class="nav">음성 채널</button>
<button data-screen="chat" class="nav">채팅 채널</button>
</div>
</div>
<div class="group" data-group="talk">
<button class="group-head" aria-expanded="false">대화 방식</button>
<div class="group-items">
<button data-screen="call" class="nav">호출</button>
<button data-screen="response" class="nav">여럿이 말할 때</button>
<button data-screen="memory" class="nav">대화 기억</button>
<button data-screen="notes" class="nav">사람별 지침</button>
</div>
</div>
<div class="group" data-group="model">
<button class="group-head" aria-expanded="false">음성·모델</button>
<div class="group-items">
<button data-screen="voice" class="nav">받아 적기·말하기</button>
<button data-screen="llm" class="nav">대화 모델</button>
</div>
</div>
<div class="group" data-group="features">
<button class="group-head" aria-expanded="false">기능</button>
<div class="group-items">
<button data-screen="features" class="nav">기능 켜기·MCP</button>
<button data-screen="schedule" class="nav">예약</button>
<button data-screen="collect" class="nav">백그라운드 수집</button>
</div>
</div>
<div class="group" data-group="pc">
<button class="group-head" aria-expanded="false">PC</button>
<div class="group-items">
<button data-screen="pcControl" class="nav">PC 제어·화면 보기</button>
<button data-screen="game" class="nav">게임 모드</button>
</div>
</div>
<div class="group" data-group="admin">
<button class="group-head" aria-expanded="false">관리</button>
<div class="group-items">
<button data-screen="logs" class="nav">로그</button>
<button data-screen="app" class="nav" id="nav-app">앱</button>
</div>
</div>
<div class="side-foot" id="version"></div>
</nav>
<main id="screen" class="screen"></main>

View File

@@ -17,6 +17,14 @@ button, input, select, textarea { font: inherit; color: inherit; }
.nav { text-align: left; background: none; border: 0; padding: 8px 10px; border-radius: 6px; color: var(--ink2); cursor: pointer; }
.nav:hover { background: var(--s2); }
.nav.on { background: var(--s3); color: var(--ink); }
.group { display: flex; flex-direction: column; margin-top: 6px; }
.group-head { text-align: left; background: none; border: 0; padding: 8px 10px; border-radius: 6px; color: var(--ink3); cursor: pointer; font-size: 12px; font-weight: 600; letter-spacing: 0.2px; display: flex; align-items: center; gap: 6px; }
.group-head::before { content: "▸"; font-size: 10px; color: var(--ink4); }
.group-head[aria-expanded="true"]::before { content: "▾"; }
.group-head:hover { background: var(--s2); color: var(--ink2); }
.group-items { display: none; flex-direction: column; gap: 2px; }
.group.open .group-items { display: flex; }
.group-items .nav { padding-left: 24px; }
.side-foot { margin-top: auto; color: var(--ink4); font-size: 12px; padding: 8px 10px; }
.screen { padding: 28px 32px; max-width: 1100px; }
h1 { font-size: 24px; font-weight: 600; letter-spacing: -0.4px; margin: 0 0 18px; }

View File

@@ -20,13 +20,14 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- **Operator instructions** (two sources, both framed "Additional instructions from the operator:" and appended near the end of the guidance list): the settings-UI `llm_instructions` config field, and every `*.md` file in `AGENTS_DIR` (default `/app/agents`, bind-mounted from `./agents`). The file-based set is read once per turn by `load_agent_instructions()` in [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) and concatenated in filename order, so dropping/editing a `.md` applies on the next reply with no rebuild/restart; fail-open to `""` when the folder is absent/empty/unreadable.
- **Warm profile block** (query-agnostic User + Directives excerpt from the knowledge graph, composed by `build_warm_profile()` / `format_warm_profile_block()` in [src/jarvis/memory/graph_ops.py](src/jarvis/memory/graph_ops.py) at Step 3.5 of `reply()`; no LLM call, pure SQLite read; injected unconditionally so personalisation is the default; result cached in `DialogueMemory._hot_cache` under `DialogueMemory.WARM_PROFILE_CACHE_KEY` for the lifetime of the active conversation. Invalidated on `stop`, on new-conversation entry, AND on User/Directives graph mutations via the listener registered in [src/jarvis/daemon.py](src/jarvis/daemon.py) against `register_graph_mutation_listener` in [src/jarvis/memory/graph.py](src/jarvis/memory/graph.py); World-branch writes are ignored)
- Digested memory enrichment (optional, see #4)
- **v2 cancel** (`turn_state.set_cancel`, set by the voice worker's brain): once the app takes a voice request back (its asker went on talking, a newer request or a stop), the loop stops before its next chat-model call or tool call and the reply returns no answer, so a dropped request never holds up the one that replaces it (`voice_worker.spec.md`, "History and brain").
- **v2 history context** (optional, caller-supplied `history=` on `run_reply_engine`): the recent window and search matches from the history store ([src/jarvis/memory/history.py](src/jarvis/memory/history.py), [spec](src/jarvis/memory/history.spec.md)), scoped to the asker and capped by the store's size budget (default about 4,000 tokens, matches at most half). No LLM call; not gated by the planner. Injected in a `<conversation_records>` fence labelled "Information shared in previous conversations"; the lines are data, not instructions. Eval: `evals/test_history_context_recall.py`. With `legacy_memory=False` (the v2 brain in the voice worker) the warm profile and the whole memory enrichment chain (#3 extractor, #3b recall gate, #4 digest and the diary/graph reads) are skipped for the turn, because v1's long-term memory belongs to a single user.
- Time + location context (re-injected each turn)
- **v2 chat channel tool**: when the voice worker's brain sets a chat backend (`turn_state.set_chat`), `chatChannels` is added to the turn's tools after routing (like `setBroadcast`) and its read/search results enter the loop inside an UNTRUSTED CHAT MESSAGES fence; without a backend it is not in the catalogue at all. Eval: `evals/test_chat_channels_tool.py`.
- **v2 schedule tool and scheduled tasks**: when the voice worker's brain sets a schedule backend (`turn_state.set_schedule`, every voice and chat reply), `schedule` is in the catalogue for the router (#7) and `prepare_intent()` (not force-offered every turn); without a backend it is not in the catalogue at all ([spec](src/jarvis/tools/builtin/schedule.spec.md)). A scheduled `task` the app fires is one more run of this loop (`reply` `via: "task"`, at the task's time, in the worker's chat lane, waiting for LLM switches like any reply): the asker's history scope, a fresh short-term memory, no prepared pick, no chat or schedule backend, `mcps={}` and `disabled_tools` widened to everything but `TASK_TOOLS` (`getWeather`, `webSearch`, `fetchWebPage`, `fetchMeals`, `collectedNews`, `toolSearchTool`), so the router, planner and loop see only those. The app waits at most 300 s for its answer and delivers it (`app/src/admin.spec.md`, "Schedules"). Eval: `evals/test_schedule_tool.py`.
- **v2 personal notes**: the app sends the asker's standing instructions with every voice, chat and task reply; `run_reply_engine(person_notes=...)` adds them to the system message (after the warm profile, before the history block) inside a `<person_notes>` fence, labelled as that person's own answer preferences that never permit an action or a tool. Only the asker's notes, at most 20 of 300 characters. The `personalNotes` tool (add, list, delete) is in the catalogue only with a notes backend (`turn_state.set_notes`, every voice and chat reply; not tasks), also for `prepare_intent()`. No extra LLM call ([spec](src/jarvis/tools/builtin/personal_notes.spec.md)). Eval: `evals/test_personal_notes.py`.
- **v2 collected news**: the `collectedNews` tool (list targets, read a target's newest items) puts the background collection's results into the loop inside an UNTRUSTED COLLECTED CONTENT fence; offered only with the app's collection file (`JARVIS_COLLECT_PATH`) and `features.collect` on, and one of the scheduled tasks' tools (`TASK_TOOLS`). No LLM call inside. The app's news notifications post titles and links with no LLM ([spec](src/jarvis/tools/builtin/collected_news.spec.md)). Eval: `evals/test_collected_news.py`.
- **v2 screen view (pictures)**: the `screenView` tool (whole screen, one monitor or one window, or the list of monitors and windows) returns base64 JPEG pictures with its result; the loop puts them on the tool message (`images`) and `llm.py` sends them to Strata as a user message of `image_url` parts right after the tool messages (about 768 / 1280 px long side, or four parts for `detailed`; at most 3 MiB). Offered only when the app makes screen viewing available (Windows, someone registered for PC control), which also starts the LLM with `--vision gpu` (normal stage) or `cpu` (game stage), else `--vision off`. Pictures stay in the one reply: not in the tool carryover, logs or prompt dumps. No extra LLM call ([spec](src/jarvis/tools/builtin/screen_view.spec.md)). Eval: `evals/test_screen_view_tool.py`.
- **v2 screen view (pictures)**: the `screenView` tool (whole screen, one monitor, one window or someone's Discord screen share in its popped-out window, or the list of monitors and windows) returns base64 JPEG pictures with its result; the loop puts them on the tool message (`images`) and `llm.py` sends them to Strata as a user message of `image_url` parts right after the tool messages (about 768 / 1280 px long side, or four parts for `detailed`; at most 3 MiB). Offered only when the app makes screen viewing available (Windows, someone registered for PC control), which also starts the LLM with `--vision gpu` (normal stage) or `cpu` (game stage), else `--vision off`. Pictures stay in the one reply: not in the tool carryover, logs or prompt dumps. No extra LLM call ([spec](src/jarvis/tools/builtin/screen_view.spec.md)). Eval: `evals/test_screen_view_tool.py`.
- Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68))
- Tool results from prior turns (raw or digested — see #5)
- **Output**: OpenAI-style `{content, tool_calls, thinking}`. Consumed by the tool orchestrator and TTS pipeline. Natural-language content is delivered immediately; no post-turn evaluator runs. Spoken-answer length: the persona (`system_prompt.py`) and `voice_style` (`prompts/system.py`) both constrain the reply to a SINGLE sentence — any dry aside must fold into that one sentence as a trailing clause, never a second sentence. This keeps TTS latency down (synth time scales with text length) and matches the `agents/llm.md` operator instruction.

View File

@@ -51,7 +51,7 @@ v1도 STT는 이미 faster-whisper입니다. 느린 진짜 이유는 엔진이
[게임PC (Windows 11) 한 대에서 전부 실행]
1. Jarvis 관리 앱 (exe, Rust. 화면은 Tauri 2 + Windows 기본 WebView2)
├─ GUI: 대시보드 / 봇 / 채널·호출 / 채팅 채널 / 대화 기억 / 모델 / 기능
├─ GUI: 대시보드 / 대화 / 디스코드 / 대화 방식 / 음성·모델 / 기능 / PC / 관리
│ / 게임 모드 / PC 제어 / 화면 보기 / 데이터 / 로그
├─ 서비스 관리자: 아래 프로세스를 띄우고, 감시하고, 죽으면 재시작
├─ 디스코드 음성 코어 (serenity + songbird + DAVE 복호화, Opus 해독은 앱이 직접)

View File

@@ -18,7 +18,7 @@
| 설정 | 내용 |
|---|---|
| 호출어(prefix) | GUI에서 여러 개 등록·수정·삭제 (예: "자비스", "야 자비스", "javis"). 음성과 채팅에 같이 쓰고, 채팅용은 따로 지정할 수도 있습니다. 비슷한 발음 허용 정도(예: "자비즈")도 조절합니다. |
| 호출어(prefix) | GUI에서 여러 개 등록·수정·삭제. 기본값은 비서를 부르듯 "비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"입니다(사용자 요청, 2026-11-19, 그 전에는 "자비스"). 한 자리에서 여러 호출어가 들리면 가장 가깝고 긴 것을 부른 말로 보므로 "비서야, 멈춰"의 "야"가 남지 않습니다. 음성과 채팅에 같이 쓰고, 채팅용은 따로 지정할 수도 있습니다. 비슷한 발음 허용 정도(예: "자비즈")도 조절합니다. "비서"는 흔한 낱말이라 "우리 회사 비서가"처럼 대화 중에 나와도 부른 것으로 봅니다. |
| 호출 방식 | (1) 호출어로 부를 때만 응답 **(기본값)** / (2) 봇에게 하는 말로 판단되면 응답 / (3) 모든 말에 응답 |
| 호출어 뒤 쉼 | "자비스, … 지금 몇 시야?"처럼 이름 뒤에 쉬어서 "자비스"만 따로 들리면, 같은 사람이 이어서 하는 말(기본 1.5초 안에 시작)과 합쳐 한 질문으로 받습니다. 아무 말도 이어지지 않으면 "자비스"에만 대답합니다. 채팅은 합치지 않습니다. |
| 지정한 사람 | GUI에서 서버 멤버 목록으로 고릅니다. 아래 두 설정을 **각각 켜고 끕니다**. |

View File

@@ -47,6 +47,8 @@
## 설정 화면
왼쪽 메뉴는 대시보드와 대화를 위에 두고, 나머지는 묶음으로 접어 둡니다(사용자 요청, 2026-11-19). 묶음 이름을 누르면 펼쳐집니다: 디스코드(봇, 음성 채널, 채팅 채널), 대화 방식(호출, 여럿이 말할 때, 대화 기억, 사람별 지침), 음성·모델(받아 적기·말하기, 대화 모델), 기능(기능 켜기·MCP, 예약, 백그라운드 수집), PC(PC 제어·화면 보기, 게임 모드), 관리(로그, 앱). 아래 표의 "채널·호출"은 음성 채널, 호출, 여럿이 말할 때 세 화면으로, "모델"은 받아 적기·말하기와 대화 모델로 나눴습니다. 화면마다 정확히 어떤 항목이 있는지는 [admin.spec.md](../../app/src/admin.spec.md) "Screens"에 있습니다.
| 화면 | 내용 |
|---|---|
| 대시보드 | 서비스 상태(디스코드, LLM, STT, TTS), GPU/VRAM/RAM 사용량, 현재 음성 채널과 참가자, 실시간 자막, 단계별 지연 시간, 성능 단계(평소/게임 중/일시정지와 이유), 놓친 요청, 봇 시작/정지 |

View File

@@ -319,6 +319,8 @@ LLM과 STT를 함께 켠 측정(M0 `gamemode_stt`, 게임 없음). STT를 먼저
최소화하면 캡처가 안 되므로 그때는 창을 다시 열어 달라고 안내합니다.
**지금 구현(M7 3단계 둘째 부분, 2026-11-19)**: `screenView`의 `share` 대상(사람 이름은 선택), 앱 `app/src/screen.rs`와 `discord.rs`([admin.spec.md](../../app/src/admin.spec.md) "Screen viewing"). 분리 창은 제목이 "Discord"로 끝나지 않는 디스코드 창으로 찾고, 자비스가 있는 음성 채널 이름이 제목에 든 창을 먼저 씁니다. 창이 없으면 음성 채널에 "디스코드에서 철수님 화면공유를 시청하고 창 분리 버튼을 눌러 주세요."라고 말하고 1초마다 30초 동안 찾습니다(채팅으로 물으면 기다리지 않고 같은 안내로 답함). 대기 시간은 아직 설정 화면에 두지 않았습니다. 실제 디스코드 분리 창 제목과 실제 모델(Qwen3.6)의 도구 고르기는 게임PC에서 확인할 것. 작은 모델(gemma3:4b, qwen3:8b)로 돌린 평가에서는 "철수 화면 뭐 하고 있어?"에 `share`를 고르는 일이 드물었습니다.
## 백그라운드 크로미움으로 정보 수집
- 봇용 크롬(데이터 폴더 안)을 크롬 개발자 프로토콜(CDP)로 조종합니다. 웹 검색, 페이지 읽기, 예약 수집에 씁니다.

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,84 @@
# STT 정확도 개선 계획 (2026-11-19)
사용자 요청(2026-11-19): 받아 적기가 문장을 제대로 맞히는 일이 드물고, 끊어 말한 문장을 하나씩 따로 처리해서 느림.
음성 인식을 고칠 때마다 다른 곳이 어긋났으니, 이번에는 원인과 판정 기준을 먼저 정하고 측정은 한 번에 끝낸 뒤 코드는 한 번만 바꿈.
모델이 약하면 바꾸되 다른 모델과 같은 자료로 비교함.
## 실제로 틀린 예 (2026-11-19 사용 중, 사용자가 준 정답)
| 들은 것 | 실제로 한 말 |
|---|---|
| 기사야 디코 화면군이 보여 | 비서야 디코 화면공유 보여? |
| 내 화면 공이 안 보여? | 내 화면 공유 안보여? |
| 실행하면 제일 위로와야지 가면 | 실행하면 제일 위로와야지 화면 |
| 지금 화면 봐봐 지금 어 이게 계산지가 제일 위에 있는 거야 | 지금 화면 봐봐 지금 어 이게 계산기가 화면 제일 위에 있는 거야? |
| 기구 어디 갔어? | 디코 어디 갔어? |
11-18 새벽 녹음 11개에서도 문장 첫머리의 호출어가 4개 틀림(아비스, 자비, 다비스, 사비스). 틀린 곳은 대부분 첫소리 자음(ㅂ→ㄱ, ㄷ→ㄱ, ㅎ→ㄱ, ㅈ→ㅇ/ㄷ/ㅅ)과 문장 끝의 짧은 낱말.
## 지금 구조 (바꿀 수 있는 곳)
- 앱이 디스코드 소리를 받아 사람마다 워커로 보냄. 100ms 동안 패킷이 없으면 `speaking_end`(디스코드는 말을 멈추면 보내기를 멈춤).
- 워커(`segmenter.py`)는 음성 확률이 0.35 아래로 `end_silence_ms`(300ms) 이어지거나 `speaking_end`가 오면 말 하나를 끝냄. 그 조각만 따로 받아 적음.
- 받아 적기: faster-whisper large-v3-turbo, int8_float16, beam 2, 언어 ko, 호출어 힌트(힌트 없는 결과로 검사).
- 앱(`conversation.rs`)은 조각마다 호출 판정. 호출어만 말한 조각만 다음 조각과 합침. 그 밖의 조각은 각각 따로 요청이 되거나 버려짐.
## 원인 가설과 확인 방법
| | 가설 | 지금까지의 근거 | 확인 방법 |
|---|---|---|---|
| H1 | 너무 짧게 자름. 0.3초만 쉬어도, 디스코드가 0.1초 보내기를 멈춰도 말이 끝나서, 짧은 조각을 문맥 없이 받아 적음. 사용자 불만 2(한 문장씩 처리)의 원인이기도 함 | 예 3·5처럼 짧은 조각이 틀림. 조각마다 따로 답함 | 오늘 워커 로그의 끝난 이유(`reason`) 비율과 같은 사람 조각 사이 간격 분포. 같은 소리를 조각마다 받아 적기 대 이어 붙여 한 번에 받아 적기 |
| H2 | 말 첫머리가 잘려 들어옴(디스코드 쪽 음성 감지가 늦게 보내기 시작) | 11-18 클립 1개(012257, "아비스")는 클립 시작 20ms에 이미 말소리. 나머지 틀린 호출어 3개는 70~180ms 여유가 있어 잘림이 아님. 일부만 설명함 | 모든 클립의 첫 소리 앞 여유와 첫 낱말 오류의 관계 |
| H3 | 모델 한계. turbo는 디코더가 4층(large-v3는 32층)이고 int8, beam 2라 짧은 구어(디코, 화면공유)에 약함 | 측정 없음 | 같은 소리로 모델 비교(아래) |
| H4 | 앞 대화 문맥이 없음 | 측정 없음 | 직전 받아 적은 문장을 프롬프트로 준 변형. 없는 말을 지어내는지 잡음 108개로 함께 확인 |
이미 확인해서 빼는 것:
- 음량: 11-16 측정, 키워도 나아지지 않음(`groups/javis_bot/logs/stt-level-2026-11-16`).
- 고음 강조: 11-18 측정, 오히려 나빠져서 0.1.36에서 되돌림.
- 소리 손상: 11-18 클립 11개의 4~7kHz 성분이 Jarvis TTS와 같은 수준(300~1000Hz 대비 -27~-32dB)이고 7.2kHz 위는 워커 필터가 자른 것. 대역이 좁아진 소리는 아님. 오늘 클립에서 20ms 구멍(패킷 손실) 수도 함께 셈.
## 시험 자료 (한 번 만들고 모든 비교에 같이 씀)
- A. 사용자 실제 녹음: 게임PC `C:\JarvisM0\Jarvis\Jarvis\recordings`의 11-18, 11-19 `said` 클립과 서버에 있는 11-18 새벽 11개. 정답은 사용자가 준 5개 말고는 후보 모델 결과를 owner와 reviewer가 따로 맞춰 보고, 둘이 다르거나 모델끼리 갈리는 것만 사용자에게 짧은 목록으로 확인.
- A-turn. A에서 같은 사람이 이어 말한 조각을 실제 시간 간격대로 이어 붙인 것(H1용).
- B. 악조건: A를 오프라인으로 바꾼 것. 소리 작게(-15dB), 생활 잡음(ESC-50) SNR 10·5dB, 싼 헤드셋 대역(300~3400Hz), 낮은 비트레이트 Opus(16kbps) 재인코딩, 방 울림. 다른 사람들은 마이크가 나쁘고 작게, 흐리게 말한다는 조건.
- C. 다른 목소리: 공개 한국어 음성(FLEURS ko, Common Voice ko, Zeroth, 가능하면 전화 음성) 200개 정도와 그 B 처리본.
- 잡음 108개(M1에서 쓴 ESC-50): 없는 말이나 호출어를 지어내는지.
## 후보 모델
| 후보 | 엔진 | 확인할 것 |
|---|---|---|
| 지금: large-v3-turbo int8_float16 beam 2 | faster-whisper | 기준 |
| large-v3 (fp16, int8) | faster-whisper | 디코더 32층. VRAM과 속도 |
| 한국어 미세조정 Whisper: BuzzASR/korean(large-v3 기반), batisay-ko-turbo, ghost613/whisper-large-v3-turbo-korean | faster-whisper로 변환 | 엔진을 그대로 쓰므로 바꾸기 쉬움. 라이선스, 영어 낱말 섞인 말, 지어내기 |
| Qwen3-ASR-1.7B와 한국어 미세조정판(TeamUNIVA/qwen3_asr_1.7b_ko_beta) | transformers | 엔진이 달라 Windows 설치, 10명 묶음 처리, VRAM을 따로 확인 |
한국어 전용 모델은 다른 언어를 못 받아 적을 수 있음. 바꾸게 되면 언어 설정이 그 모델의 언어일 때만 쓰고, 다른 언어는 지금 모델을 씀(코드에 언어별 규칙은 넣지 않음).
## 판정 기준 (측정 전에 정함)
1. A와 B의 문자 오류율(띄어쓰기·문장부호 빼고, 숫자는 읽는 대로), 호출어 놓침, 잡음에서 지어낸 호출어.
2. 조건: RTX 5070 Ti에서 말 한 마디의 최종 전사 p95 0.4초 이하(지금 0.11~0.19초, pc-features.md). 10명 동시 전사 VRAM이 지금 STT 몫(최대 약 2.7GB) 안. 넘으면 LLM 배치 식을 다시 재야 하므로 사용자에게 물음.
3. 기준보다 A+B 오류율이 상대 20% 넘게 낮을 때만 모델을 바꿈. 자료가 적어 그보다 작은 차이는 우연과 구분할 수 없음. C는 동률일 때만 봄.
4. 말 차례 합치기(아래)는 A-turn에서 한 번에 받아 적은 쪽 오류율이 낮고, 늘어나는 대기가 사용자가 정한 범위 안일 때 넣음.
## 끊어 말한 문장 합치기 (불만 2, 스펙 변경이라 사용자 확인 필요)
제안: 같은 사람이 짧게 쉬었다 다시 말하면 같은 "말 차례"로 봄. 차례가 끝날 때(쉰 시간이 `end_silence_ms`를 넘을 때) 그 소리 전체를 한 번에 받아 적어 요청 하나로 처리. 디스코드가 보내기를 멈춘 것(`speaking_end`)은 바로 끝내지 않고 쉰 시간으로 셈.
- 좋아지는 것: 여러 문장을 한 질문으로 받아 답 한 번. 받아 적기가 문맥을 가짐(H1).
- 대가: 한 문장만 말해도 끝을 알기까지 그만큼 더 기다림. 말하는 동안의 부분 전사와 답 준비(`prepare`)는 지금처럼 돌아서 일부는 겹쳐짐.
- 기다릴 시간은 A의 실제 간격 분포를 보고 정함(문장 사이 쉼의 대부분을 덮는 값). 1초를 넘으면 사용자에게 다시 물음.
- 바뀌는 스펙: `voice_worker.spec.md`(말 끝내기), `conversation.spec.md`(호출어만 말한 경우 합치기는 대부분 필요 없어짐), `discord_core.spec.md`(`speaking_end`의 뜻).
## 순서
1. 자료 모으기: 게임PC에서 녹음과 워커 로그 복사(읽기만, 사용자가 정한 시간에).
2. 진단: H1~H4 측정, 결과를 이 문서에 적고 reviewer가 교차 검토.
3. 모델 비교: 게임PC GPU에서 한 번(Jarvis 서비스를 끄고), 고른 모델은 Strata와 함께 켠 상태로 VRAM·지연 한 번.
4. 결정한 것만 한 번에 구현(모델, 말 차례), 테스트, 스펙 수정.
5. 같은 시험 자료로 최종 확인 후 릴리스 한 번, 사용자 실사용 확인 한 번.
## 진행 기록
- 2026-11-19: 서버 GPU 사용 불가. 15:19 재부팅으로 커널 7.0.0-43이 올라왔는데 이 커널용 nvidia 모듈 패키지(`linux-modules-nvidia-595-open-7.0.0-43-generic`)가 설치되지 않음. 서버 측정은 CPU로만 가능.

View File

@@ -6,6 +6,8 @@ Eval: seeing the screen (src/jarvis/tools/builtin/screen_view.spec.md).
- When the app refuses (the person is not registered), the answer says it cannot see the screen and
does not describe one.
- A question that is not about the screen does not capture it.
- "철수 화면 뭐 하고 있어?" asks for 철수's Discord screen share (target share, person 철수), and
the answer reads that picture.
The screen backend is a fake that returns a fixed picture; the tool, the reply loop and the picture
transport (llm.py) run for real. Needs a model that reads pictures:
@@ -38,8 +40,8 @@ class FakeScreen:
{"number": 2, "width": 1280, "height": 720, "main": False}],
"windows": [{"program": "notepad", "title": "Untitled - Notepad",
"minimised": False}]}}
return {"ok": True, "item": {"what": f"모니터 {kw.get('monitor', 2)}", "width": 1280, "height": 720,
"parts": 1}, "images": [PICTURE]}
what = "화면공유" if kw.get("target") == "share" else f"모니터 {kw.get('monitor', 2)}"
return {"ok": True, "item": {"what": what, "width": 1280, "height": 720, "parts": 1}, "images": [PICTURE]}
def _run(mock_config, eval_db, eval_dialogue_memory, query, screen):
@@ -84,3 +86,11 @@ class TestScreenView:
screen = FakeScreen()
_run(mock_config, eval_db, eval_dialogue_memory, "고양이는 하루에 몇 시간 자?", screen)
assert not any(a == "capture" for a, _ in screen.acts), screen.acts
def test_someones_screen_share_is_asked_for_by_their_name(self, mock_config, eval_db, eval_dialogue_memory):
screen = FakeScreen()
reply = _run(mock_config, eval_db, eval_dialogue_memory, "철수 화면 뭐 하고 있어?", screen)
captures = [kw for a, kw in screen.acts if a == "capture"]
assert captures and captures[0].get("target") == "share" and "철수" in (captures[0].get("person") or ""), \
screen.acts
assert "BANANA" in reply.upper() or "바나나" in reply or "42" in reply, reply

View File

@@ -2376,6 +2376,9 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
_one_step_nudged = False
while turn < max_turns:
if _turn_state.cancelled():
debug_log("reply cancelled: no longer wanted", "planning")
return None
turn += 1
debug_log(f"🔁 messages loop turn {turn}", "planning")
print(f" 🔁 Turn {turn}/{max_turns}", flush=True)
@@ -2840,7 +2843,10 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
messages.append({"role": "tool", "tool_call_id": tool_call_id, "content": f"You have already called {tool_name} {duplicate_tool_count} times. Please use the results from those calls to answer the user's question."})
continue
# Execute tool
# Execute tool (not for an answer that is no longer wanted: no actions on its behalf)
if _turn_state.cancelled():
debug_log("reply cancelled before a tool call", "planning")
return None
tools_started += 1
result = run_tool_with_retries(
db=db,

View File

@@ -37,6 +37,9 @@ pc : Optional[object]
trace : Optional[list]
v2: where the engine writes down how it made this answer (history.spec.md, "Traces"): the
tools it was given, the plan, the model's thinking and each tool call. None records nothing.
cancel : Optional[callable]
v2: true once the answer is no longer wanted (the asker went on talking, or a newer request or a
stop took its place). The engine checks it before each model call and each tool call.
"""
from __future__ import annotations
@@ -58,6 +61,17 @@ def reset() -> None:
_state.screen = None
_state.pc = None
_state.trace = None
_state.cancel = None
def set_cancel(check) -> None:
_state.cancel = check
def cancelled() -> bool:
"""Whether this turn's answer is no longer wanted."""
check = getattr(_state, "cancel", None)
return bool(check and check())
def set_trace(steps: Optional[list]) -> None:

View File

@@ -10,9 +10,10 @@ from ..base import Tool, ToolContext
from ..types import ToolExecutionResult
from ...debug import debug_log
_TARGETS = ("screen", "monitor", "window", "list")
_TARGETS = ("screen", "monitor", "window", "share", "list")
_DETAILS = ("fast", "normal", "detailed")
MAX_WINDOW_CHARS = 200
MAX_PERSON_CHARS = 200
def _fail(text: str) -> ToolExecutionResult:
@@ -37,10 +38,11 @@ class ScreenViewTool(Tool):
def description(self) -> str:
return (
"See this PC's screen as a picture: target='screen' (all monitors), 'monitor' with its "
"number, or 'window' with part of the program's name or window title; target='list' "
"names the monitors and open windows. Use it whenever the person asks what is on their "
"screen, a monitor or a program's window; detail='detailed' to read small text. Answer "
"only from the picture."
"number, 'window' with part of the program's name or window title, or 'share' for "
"another person's screen, which they share on Discord (person: their name); "
"target='list' names the monitors and open windows. Use it whenever the person asks "
"what is on their screen, a monitor, a program's window or someone else's screen; "
"detail='detailed' to read small text. Answer only from the picture."
)
@property
@@ -51,6 +53,7 @@ class ScreenViewTool(Tool):
"target": {"type": "string", "enum": list(_TARGETS)},
"monitor": {"type": "integer", "description": "monitor: its number (1, 2, ...)"},
"window": {"type": "string", "description": "window: part of its program name or title"},
"person": {"type": "string", "description": "share: the name of the person sharing"},
"detail": {"type": "string", "enum": list(_DETAILS)},
},
"required": [],
@@ -80,6 +83,12 @@ class ScreenViewTool(Tool):
if not w or len(w) > MAX_WINDOW_CHARS:
return _fail("window needs part of the program name or title; call target='list' to see them")
fields["window"] = w
if target == "share":
p = " ".join(str(args.get("person") or "").split())
if len(p) > MAX_PERSON_CHARS:
return _fail("person needs the name of the person sharing their screen")
if p:
fields["person"] = p
fields["detail"] = _word(args.get("detail"), _DETAILS, "normal")
result = backend.act("capture", **fields) or {}
images = [i for i in result.get("images") or [] if isinstance(i, str) and i]

View File

@@ -32,6 +32,7 @@ Lets Jarvis see this PC's screen: all monitors as one picture, one monitor, or o
| `screen` (default) | | every monitor as one picture |
| `monitor` | `monitor` (its number, as Windows numbers it) | that monitor |
| `window` | `window` (part of its title or program name) | that window, also when other windows cover it |
| `share` | `person` (optional: the name of the person sharing, as said) | the Discord screen share someone streams in Jarvis's voice channel, from the window Discord pops it out to |
| `list` | | the monitors (number, size, main) and the open windows (program, title), no picture |
`detail`: `fast` (long side about 768 px), `normal` (default, about 1280 px) or `detailed` (the
@@ -44,6 +45,9 @@ as their first known word; unknown values fall back to the defaults.
- `act("capture", target=..., monitor=..., window=..., detail=...)` gives `{"ok": true, "item":
{"what", "width", "height", "parts"}, "images": ["<base64 JPEG>", ...]}`.
- `act("capture", target="share", person=..., detail=...)` gives the same; `person` is sent only
when given (at most 200 characters). The app checks that the person shares their screen and may
wait up to 30 s while it asks for the share to be popped out (admin.spec.md, "Screen viewing").
- `act("list")` gives `{"ok": true, "item": {"monitors": [...], "windows": [...]}}`.
- Failures are `{"ok": false, "error": "..."}`.

View File

@@ -10,7 +10,7 @@ import unicodedata
import threading
import time
from dataclasses import dataclass
from typing import Optional, Protocol
from typing import Callable, Optional, Protocol
from .log import debug_log, notice
from jarvis.memory.history import HistoryContext, Scope
@@ -19,7 +19,8 @@ from jarvis.memory.history import HistoryContext, Scope
class Brain(Protocol):
def reply(self, user: str, text: str, history: HistoryContext, chat=None, schedule=None,
task: bool = False, notes: Optional[list] = None, notes_backend=None,
screen=None, pc=None, trace: Optional[list] = None) -> Optional[str]:
screen=None, pc=None, trace: Optional[list] = None,
cancelled: Optional[Callable[[], bool]] = None) -> Optional[str]:
"""Blocking; called on a worker thread, one call per person at a time (a voice and a chat
reply for different people may overlap). ``chat``: the chat backend for
the chatChannels tool (voice_worker/chat_bridge.py), or None without registered channels.
@@ -28,7 +29,8 @@ class Brain(Protocol):
``notes``: the asker's standing instructions, put in the prompt; ``notes_backend``: the
backend for the personalNotes tool (personal_notes.spec.md); ``screen``: the backend for the
screenView tool (screen_view.spec.md); ``trace``: a list the engine adds the steps of
this answer to (history.spec.md, "Traces")."""
this answer to (history.spec.md, "Traces"); ``cancelled``: true once the answer is no longer
wanted, so the engine stops at its next model or tool call and None is returned."""
def addressed(self, user: str, text: str, names: list[str], people: Optional[int] = None,
callers: Optional[int] = None) -> Optional[bool]:
@@ -310,7 +312,8 @@ class JarvisBrain:
def reply(self, user: str, text: str, history: HistoryContext, chat=None, schedule=None,
task: bool = False, notes: Optional[list] = None, notes_backend=None,
screen=None, pc=None, trace: Optional[list] = None) -> Optional[str]:
screen=None, pc=None, trace: Optional[list] = None,
cancelled: Optional[Callable[[], bool]] = None) -> Optional[str]:
from jarvis.memory.conversation import DialogueMemory
from jarvis.reply import turn_state
from jarvis.reply.engine import run_reply_engine
@@ -334,6 +337,7 @@ class JarvisBrain:
turn_state.set_screen(None if task else screen)
turn_state.set_pc(None if task else pc)
turn_state.set_trace(trace)
turn_state.set_cancel(cancelled)
try:
return run_reply_engine(self.db, cfg, None, text, memory,
language=self.language, history=history, legacy_memory=False,

View File

@@ -169,8 +169,9 @@ class VoiceWorker:
def action_timeout_s(self, kind: str) -> float:
"""How long an app action is waited for: a PC action (starting a program and reading its new
window, admin.spec.md "PC control") gets at least PC_ACTION_TIMEOUT_S."""
return max(self.chat_timeout_s, PC_ACTION_TIMEOUT_S) if kind == "pc" else self.chat_timeout_s
window, admin.spec.md "PC control") and a screen action (a screen share waited for while it
is popped out, "Screen viewing") get at least PC_ACTION_TIMEOUT_S."""
return max(self.chat_timeout_s, PC_ACTION_TIMEOUT_S) if kind in ("pc", "screen") else self.chat_timeout_s
async def start(self) -> None:
if self.history is not None:
@@ -280,7 +281,7 @@ class Session:
self.tts_recorded: dict[str, dict] = {}
# Spoken replies whose speech was cancelled while the brain was still answering.
self.cancelled_replies: set[str] = set()
self.thinking_replies: set[str] = set()
self.thinking_replies: dict[str, str] = {} # reply id -> asker
# user -> the newest call text waiting for that user's running prepare to finish (or None).
self.preparing: dict[str, str | None] = {}
self.last_prepared: dict[str, str] = {} # user -> the text of their last prepare
@@ -680,12 +681,15 @@ class Session:
asyncio.get_running_loop().create_task(self._task(rid, user, text, notes))
return
if speak:
self.thinking_replies.add(rid)
self.thinking_replies[rid] = user
if user in self.preparing:
self.preparing[user] = None # the call was answered: no newer text to prepare
self.last_prepared.pop(user, None)
waiting = self.worker.replies_unanswered
early = waiting.get(user, 0) > 0 # sent before their previous answer: not a reply to it
# Sent before their previous answer: not a reply to it. An answer taken back (cancelled while
# the brain thinks) does not count: what comes now replaces it.
taken_back = sum(1 for r in self.cancelled_replies if self.thinking_replies.get(r) == user)
early = waiting.get(user, 0) - taken_back > 0
waiting[user] = waiting.get(user, 0) + 1
asyncio.get_running_loop().create_task(self._reply(rid, user, text, via, speak, early, channel, notes))
@@ -917,7 +921,7 @@ class Session:
await self._reply_inner(rid, user, text, via, speak, early, channel, notes or [], trace=trace)
finally:
trace.finish("") # only when it did not get to its answer
self.thinking_replies.discard(rid)
self.thinking_replies.pop(rid, None)
self.cancelled_replies.discard(rid)
if not self._handed_over.pop(rid, False): # else _delivered counts it
self._reached(user)
@@ -985,23 +989,39 @@ class Session:
pc = AppBridge(self, asyncio.get_running_loop(), "pc", *origin, turn=turn,
state=w.pc_state.setdefault(user, {}))
trace.append(self._records_step(history, notes))
try:
async with self._llm(rid if speak else None), self._card_watch(f"answer {rid}"):
w.answering += 1
try:
answer = await asyncio.to_thread(w.brain.reply, user, text, history, chat=chat,
schedule=schedule, notes=notes or [],
notes_backend=notes_backend, screen=screen, pc=pc,
trace=trace)
finally:
w.answering -= 1
if not w.answering and w.scheduler:
w.scheduler.wake() # held partials go on
except Exception as e: # the brain must never take the session down
debug_log(f"brain reply {rid} failed: {type(e).__name__}", "llm")
answer = None
answer = None
if speak and rid in self.cancelled_replies:
# Taken back while it waited for the asker's previous answer: never started.
debug_log(f"brain reply {rid} not started: cancelled while waiting", "llm")
else:
try:
async with self._llm(rid if speak else None), self._card_watch(f"answer {rid}"):
w.answering += 1
try:
answer = await asyncio.to_thread(
w.brain.reply, user, text, history, chat=chat, schedule=schedule,
notes=notes or [], notes_backend=notes_backend, screen=screen, pc=pc,
trace=trace, cancelled=lambda: rid in self.cancelled_replies)
finally:
w.answering -= 1
if not w.answering and w.scheduler:
w.scheduler.wake() # held partials go on
except Exception as e: # the brain must never take the session down
debug_log(f"brain reply {rid} failed: {type(e).__name__}", "llm")
answer = None
answer = (answer or "").strip()
trace.finish(answer)
if not answer and speak and rid in self.cancelled_replies:
# Taken back before it was written: as if never asked, so the request that replaces
# it is still the very next turn (a confirmation it carries does not lapse).
w.reply_turns[user] = turn - (2 if early else 1)
debug_log(f"brain reply {rid} taken back", "llm")
self.send({"type": "reply_text", "id": rid, "text": "", "failed": True})
if rid in self.tts_requests:
self._on_tts({"type": "tts", "id": rid, "text": "", "more": False})
else:
self.send({"type": "tts_done", "id": rid, "chunks": 0, "cancelled": True, "failed": False})
return
if not answer and rid in self.tts_requests:
self._on_tts({"type": "tts", "id": rid, "text": "", "more": False}) # ends the notice's speech
self.send({"type": "reply_text", "id": rid, "text": answer, "failed": not answer})

View File

@@ -257,8 +257,15 @@ which carry Discord's own time.
answer is stored when the app reports the sent message as `chat_message` with `to`. With
`speak: true` the answer is also spoken through the TTS path under the same id (audio frames and
`tts_done`), and `tts_cancel` with that id stops it. A `tts_cancel` that arrives while the brain is
still answering means the answer is not spoken: `reply_text` still comes, then `tts_done` with
`cancelled: true` and no audio. Every reply, a task's too and one that failed, leaves a trace of
still answering means the answer is no longer wanted (the app took the request back: its asker
went on talking, a newer request or a stop took its place). A reply still waiting for the asker's
previous answer is never started; a running one stops at the engine's next model call or tool
call (`turn_state.cancel`: no tool runs on its behalf) and returns no answer. Such a reply is as if
never asked: `reply_text` comes with empty text and `failed: true`, then `tts_done` with
`cancelled: true` and no audio, and it uses up no turn of the asker's (the request that replaces it
is still the very next turn, so a confirmation it carries does not lapse; nor is that request
"sent before the previous answer"). A reply whose answer was already written when the cancel
came still sends its text, then `tts_done` with `cancelled: true` and no audio. Every reply, a task's too and one that failed, leaves a trace of
how it was made (`history.spec.md`, "Traces"): the history lines and number of notes it was given,
then what the engine wrote down. It is stored as soon as the request arrives (before it waits for
the asker's previous answer) and again as each step is added, then ends with the answer, or with
@@ -436,7 +443,8 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
```
<- {"type": "screen_action", "id": "s1", "action": "capture|list", "user": "<id>", "via": "voice|chat",
"channel": "...", "target": "screen|monitor|window", "monitor": 2, "window": "chrome", "detail": "fast|normal|detailed"}
"channel": "...", "target": "screen|monitor|window|share", "monitor": 2, "window": "chrome", "person": "철수",
"detail": "fast|normal|detailed"}
-> {"type": "screen_action_result", "id": "s1", "ok": true, "item": {...}, "images": ["<base64 JPEG>"]} // or ok false with "error"
```
@@ -444,6 +452,8 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
waiting are those of chat and schedule actions. `images` that are not a list of strings are
dropped. The pictures stay in that one reply (screen_view.spec.md).
- Messages may be up to 4 MiB (pictures), the WebSocket's limit.
- A screen action is waited for as long as a PC action (45 s): the app may wait up to 30 s for a
screen share to be popped out.
### PC control (M7)

View File

@@ -0,0 +1,66 @@
"""An answer that is no longer wanted stops at the engine's next model or tool call (turn_state.cancel).
The voice worker sets it when the app takes the request back (the asker went on talking and the
request is joined to what they said next, a newer request took its place, or a stop). The engine
then returns no answer and runs no tool on the request's behalf.
"""
from __future__ import annotations
from unittest.mock import patch
import pytest
def _native_call(name: str, args: dict) -> dict:
return {"message": {"role": "assistant", "content": "", "tool_calls": [
{"id": "call_1", "type": "function", "function": {"name": name, "arguments": args}}]}}
def _run(mock_config, db, dialogue_memory, cancel_after_calls: int):
from jarvis.memory.history import HistoryContext
from jarvis.reply import engine as engine_mod
from jarvis.reply import turn_state
from jarvis.tools.types import ToolExecutionResult
mock_config.ollama_chat_model = "gpt-oss:20b"
mock_config.evaluator_enabled = False
mock_config.stream_browser = False
calls = []
invoked = []
def fake_chat(*args, **kwargs):
calls.append(1)
return _native_call("getWeather", {"location": "Seoul"})
def fake_tool(db, cfg, tool_name, tool_args, **kwargs):
invoked.append(tool_name)
return ToolExecutionResult(success=True, reply_text="Seoul: 13C", error_message=None)
turn_state.reset()
turn_state.set_cancel(lambda: len(calls) >= cancel_after_calls)
try:
with patch.object(engine_mod, "run_tool_with_retries", side_effect=fake_tool), \
patch.object(engine_mod, "chat_with_messages", side_effect=fake_chat), \
patch.object(engine_mod, "select_tools", return_value=["getWeather", "stop"]), \
patch.object(engine_mod, "plan_query", return_value=["Reply to the user."]):
reply = engine_mod.run_reply_engine(
db=db, cfg=mock_config, tts=None, text="서울 날씨 어때?",
dialogue_memory=dialogue_memory, history=HistoryContext(), legacy_memory=False,
intent_judge=True)
finally:
turn_state.reset()
return reply, len(calls), invoked
@pytest.mark.unit
def test_an_answer_cancelled_before_it_starts_makes_no_model_call(mock_config, db, dialogue_memory):
reply, calls, invoked = _run(mock_config, db, dialogue_memory, cancel_after_calls=0)
assert reply is None and calls == 0 and invoked == []
@pytest.mark.unit
def test_an_answer_cancelled_during_a_model_call_runs_no_tool_for_it(mock_config, db, dialogue_memory):
# The model asks for a tool, but the answer was taken back meanwhile: the tool is not run.
reply, calls, invoked = _run(mock_config, db, dialogue_memory, cancel_after_calls=1)
assert reply is None and calls == 1 and invoked == []

View File

@@ -68,6 +68,15 @@ def test_a_window_is_found_by_part_of_its_name(app):
assert app.calls[-1] == ("capture", {"target": "window", "window": "크롬", "detail": "normal"})
@pytest.mark.unit
def test_a_discord_screen_share_is_asked_for_with_the_person_named(app):
assert run(target="share", person=" 철수 ").success
assert run(target="share").success
assert app.calls == [("capture", {"target": "share", "person": "철수", "detail": "normal"}),
("capture", {"target": "share", "detail": "normal"})]
assert not run(target="share", person="x" * 201).success and len(app.calls) == 2
@pytest.mark.unit
@pytest.mark.parametrize("args", [{"target": "monitor"}, {"target": "monitor", "monitor": 0},
{"target": "monitor", "monitor": "two"}, {"target": "window"},

View File

@@ -29,8 +29,9 @@ class FakeBrain:
self.calls = []
def reply(self, user, text, history, chat=None, schedule=None, task=False, notes=None, notes_backend=None,
screen=None, pc=None, trace=None):
screen=None, pc=None, trace=None, cancelled=None):
self.calls.append((user, text, history))
self.cancelled = cancelled
if trace is not None:
trace.append({"kind": "tools", "names": ["getWeather"]})
self.chat, self.schedule, self.task = chat, schedule, task
@@ -730,6 +731,83 @@ def test_a_reply_cancelled_while_the_brain_thinks_is_not_spoken(store):
assert replies == ["r1", "r2"], "the text still comes back; only the speech is skipped"
@pytest.mark.unit
def test_a_cancelled_reply_stops_the_brain_and_one_still_waiting_never_reaches_it(store):
"""A request taken back (the asker went on talking) must not hold up the next one: the brain is
told to stop the running answer, and a reply still waiting for it is never started."""
import time
started = []
def answer(user, text):
started.append(text)
if text == "first":
t0 = time.monotonic()
while not brain.cancelled() and time.monotonic() - t0 < 10:
time.sleep(0.01)
return None if brain.cancelled() else "late answer"
return f"answer for {text}"
brain = FakeBrain(answer)
async def body(url):
async with connect(url) as ws:
await hello(ws)
await send(ws, type="voice_join", channel="vc1")
await send(ws, type="reply", id="r1", user="100", text="first", via="voice", speak=True)
await asyncio.sleep(0.1)
await send(ws, type="reply", id="r2", user="100", text="second", via="voice", speak=True)
await send(ws, type="tts_cancel", id="r1")
await send(ws, type="tts_cancel", id="r2")
await send(ws, type="reply", id="r3", user="100", text="first second", via="voice", speak=True)
t0 = time.monotonic()
seen = await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done"
and m["id"] == "r3")
return seen, time.monotonic() - t0
seen, waited = asyncio.run(with_worker(body, store=store, brain=brain, tts=FakeTts()))
assert waited < 3, "the joined request did not wait for the cancelled answer to run out"
assert started == ["first", "first second"]
done = {m["id"]: m for m in seen if isinstance(m, dict) and m.get("type") == "tts_done"}
assert done["r1"]["cancelled"] and done["r2"]["cancelled"] and not done["r3"]["cancelled"]
frames = {decode_tts_frame(m)[0] for m in seen if isinstance(m, bytes)}
assert frames == {"r3"}
@pytest.mark.unit
def test_a_request_taken_back_does_not_use_up_the_turn_a_confirmation_needs(store):
"""Jarvis asked something; the asker says "yes" and goes on ("yes, quickly"). The app takes the
"yes" back and sends the joined text: it must still be the very next turn, or a confirmation
(pcControl, chat delete) would lapse."""
import time
turns = {}
def answer(user, text):
turns[text] = brain.pc.turn
if text == "yes":
t0 = time.monotonic()
while not brain.cancelled() and time.monotonic() - t0 < 10:
time.sleep(0.01)
return None
return f"answer for {text}"
brain = FakeBrain(answer)
async def body(url):
async with connect(url) as ws:
await hello(ws)
await send(ws, type="voice_join", channel="vc1")
await send(ws, type="reply", id="r1", user="100", text="open the calculator", via="voice", speak=False)
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
await send(ws, type="reply", id="r2", user="100", text="yes", via="voice", speak=True)
await asyncio.sleep(0.1)
await send(ws, type="tts_cancel", id="r2")
await send(ws, type="reply", id="r3", user="100", text="yes quickly", via="voice", speak=True)
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "tts_done" and m["id"] == "r3")
asyncio.run(with_worker(body, store=store, brain=brain, tts=FakeTts()))
assert turns["yes quickly"] == turns["open the calculator"] + 1
@pytest.mark.unit
def test_the_brains_judgements_use_its_model_and_map_back_to_ids(monkeypatch):
import voice_worker.judges as judges
@@ -1652,26 +1730,32 @@ def test_a_pc_action_goes_to_the_app_and_an_ask_answer_keeps_what_to_ask(store):
@pytest.mark.unit
def test_a_pc_action_waits_longer_than_the_other_app_actions(store):
def test_pc_and_screen_actions_wait_longer_than_the_other_app_actions(store):
"""Starting a program and reading its new window takes the app up to about 20 s (admin.spec.md,
"PC control"): a PC action is not given up after the chat actions' wait."""
"PC control"), and a screen share is waited for up to 30 s while it is popped out ("Screen
viewing"): those actions are not given up after the chat actions' wait."""
results = []
brain = FakeBrain(lambda u, t: results.append((brain.screen.act("list"), brain.pc.act("run", program="calc")))
or "ok")
brain = FakeBrain(lambda u, t: results.append((brain.notes_backend.act("list"),
brain.screen.act("capture", target="share"),
brain.pc.act("run", program="calc"))) or "ok")
async def body(url):
async with connect(url) as ws:
await hello(ws)
await send(ws, type="reply", id="r1", user="100", text="q", via="voice", speak=False)
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "screen_action")
seen = await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "pc_action")
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "note_action")
seen = await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "screen_action")
await asyncio.sleep(0.6) # longer than chat_timeout_s
await send(ws, type="screen_action_result", id=seen[-1]["id"], ok=True, item={"what": "화면공유"},
images=["x"])
seen = await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "pc_action")
await asyncio.sleep(0.6)
await send(ws, type="pc_action_result", id=seen[-1]["id"], ok=True, item={"window": {"id": "w1"}})
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
asyncio.run(with_worker(body, store=store, brain=brain, chat_timeout_s=0.3))
(screen, pc), = results
assert not screen["ok"] and pc == {"ok": True, "item": {"window": {"id": "w1"}}}
(note, screen, pc), = results
assert not note["ok"] and screen["ok"] and pc == {"ok": True, "item": {"window": {"id": "w1"}}}
@pytest.mark.unit