Compare commits
6 Commits
app-v0.1.4
...
app-v0.1.4
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e32671f871 | ||
|
|
e23a962923 | ||
|
|
716b5bb2e6 | ||
|
|
e88c94d934 | ||
|
|
4906d8e7dd | ||
|
|
844958d311 |
@@ -18,7 +18,7 @@
|
||||
- **빠른 음성 대화**: 말하는 도중에 전사하고 답은 문장 단위로 바로 읽어서, 말이 끝난 뒤 첫 소리까지 약 1~1.5초를 목표로 합니다.
|
||||
- **최대 10명 동시 대화**: 화자별로 따로 받아 전사합니다. 여러 명이 동시에 부를 때 어떻게 답할지(처음 대화만 / 들어온 순서대로 / 마지막 대화만 / 제일 중요한 대화만)는 GUI에서 고릅니다.
|
||||
- **호출어와 지정한 사람**:
|
||||
- 호출어는 GUI에서 여러 개 등록합니다.
|
||||
- 호출어는 GUI에서 여러 개 등록합니다. 기본값은 "비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"입니다.
|
||||
- "지정한 사람만 응답"과 "지정한 사람은 호출어 없이 대답"을 각각 켜고 끕니다.
|
||||
- **AI 채팅 채널**: 등록한 디스코드 채팅 채널을 자비스가 읽고, 메시지를 보내고, 자기 메시지를 고칩니다. 채널 ID, 이름, 설명은 GUI에서 편집하고, 고친 이름·설명은 실제 디스코드 채널 이름·주제에도 반영합니다(끌 수 있음).
|
||||
- **예약 알림과 정기 작업**: "10분 뒤에 알려 줘", "내일 7시에 회의 알려 줘", "평일 아침 9시에 날씨 알려 줘"처럼 말이나 채팅으로 맡깁니다. 맡긴 사람이 자비스와 같은 음성 채널에 있으면 음성으로, 아니면 채팅 채널에 글로 전합니다. 정기 작업은 날씨, 웹 검색처럼 찾아보는 기능만 쓰고, 관리 앱 "예약" 화면에서 목록을 보고 지울 수 있습니다.
|
||||
|
||||
2
app/Cargo.lock
generated
2
app/Cargo.lock
generated
@@ -1520,7 +1520,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.44"
|
||||
version = "0.1.46"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.44"
|
||||
version = "0.1.46"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"
|
||||
|
||||
4
app/shell/Cargo.lock
generated
4
app/shell/Cargo.lock
generated
@@ -2637,7 +2637,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-app"
|
||||
version = "0.1.44"
|
||||
version = "0.1.46"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64 0.22.1",
|
||||
@@ -2666,7 +2666,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.44"
|
||||
version = "0.1.46"
|
||||
dependencies = [
|
||||
"jarvis-app",
|
||||
"serde_json",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "jarvis-shell"
|
||||
version = "0.1.44"
|
||||
version = "0.1.46"
|
||||
edition = "2021"
|
||||
rust-version = "1.85"
|
||||
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"
|
||||
|
||||
@@ -194,6 +194,21 @@ one window; the app captures and decides who may.
|
||||
`window` takes the visible top-level window whose title or program name contains the text
|
||||
(case-insensitive; the one nearest the front when several do), also when others cover it; a
|
||||
minimised window is refused ("최소화되어 있어 볼 수 없습니다").
|
||||
- **Screen share** (`share`, `docs/v2/pc-features.md` "다른 사람의 디스코드 화면공유 보기"):
|
||||
someone's Discord screen share, seen in the window Discord pops it out to; no account or token is
|
||||
touched and Jarvis never clicks Discord. Only while Jarvis is in a voice channel ("자비스가 음성
|
||||
채널에 있을 때만 화면공유를 볼 수 있습니다") and someone there other than Jarvis shares their
|
||||
screen (`self_stream`): with a `person`, the sharer whose display name contains it or is contained
|
||||
in it (letters and digits, any case), else "<person>님은 지금 화면공유 중이 아닙니다"; with none,
|
||||
anyone sharing ("지금 음성 채널에서 화면공유 중인 사람이 없습니다" when nobody does). The window
|
||||
is a visible Discord window (`discord.exe`, `discordptb.exe`, `discordcanary.exe`) whose title does
|
||||
not end with "Discord" (the main window's does), the one whose title contains the voice channel's
|
||||
name first, else the one nearest the front; Discord on the exclusion list is never seen. A
|
||||
minimised one is refused ("창을 다시 열어 주세요"). With none, a voice request says "디스코드에서
|
||||
<name>님 화면공유를 시청하고 창 분리 버튼을 눌러 주세요." in the channel (without a name when
|
||||
several share) and looks again every second for 30 s, then refuses ("화면공유 창이 분리되지 않아
|
||||
볼 수 없습니다"); a chat request is refused at once with that instruction. Who may see is decided
|
||||
before anything is said. The worker waits 45 s for a screen action ("PC control" timing).
|
||||
- **Exclusions**: `pcControl.screenExclude` (program file names, ".exe" may be left out; default
|
||||
`1Password`, `Bitwarden`, `KeePass`, `KeePassXC`). Their windows are never captured or listed, and
|
||||
in a monitor or whole-screen picture their area is painted black before anything leaves the
|
||||
@@ -212,8 +227,8 @@ one window; the app captures and decides who may.
|
||||
saved, never logged.
|
||||
- **List**: monitors `{number, width, height, main}` and visible windows `{program, title,
|
||||
minimised}`, front first, at most 30, without excluded programs and the app's own windows.
|
||||
- **Activity**: each capture is `screen.view` with the target (`전체 화면`, `모니터 <n>`, `창`; never
|
||||
a title), who asked and whether it worked ("Activity").
|
||||
- **Activity**: each capture is `screen.view` with the target (`전체 화면`, `모니터 <n>`, `창`,
|
||||
`화면공유`; never a title or a name), who asked and whether it worked ("Activity").
|
||||
- Logs carry the action, target kind, size and time taken, never a title or a picture.
|
||||
|
||||
## PC control (`app/src/pc.rs`)
|
||||
@@ -870,8 +885,13 @@ its data folder, and again after each one is saved. The log hub writes to the da
|
||||
## Screens (`app/ui/`)
|
||||
|
||||
Plain HTML, CSS and an ES module, no build step, dark theme after `DESIGN.md`. All text is Korean.
|
||||
The side menu has 대화 right under 대시보드 and 앱 last; there is no separate update screen (updates
|
||||
are on the 앱 screen) and no separate 한 일 screen (it is a view of the 로그 screen).
|
||||
The side menu has 대시보드 and 대화 on top, then groups that open and close when their name is
|
||||
pressed (user request, 2026-11-19): 디스코드 (봇, 음성 채널, 채팅 채널), 대화 방식 (호출, 여럿이 말할
|
||||
때, 대화 기억, 사람별 지침), 음성·모델 (받아 적기·말하기, 대화 모델), 기능 (기능 켜기·MCP, 예약,
|
||||
백그라운드 수집), PC (PC 제어·화면 보기, 게임 모드) and 관리 (로그, 앱). The group of the screen
|
||||
shown opens (and the 관리 group when a new version renames the 앱 button); which groups are open is
|
||||
kept in the page's storage for the next start. There is no separate update screen (updates are on
|
||||
the 앱 screen) and no separate 한 일 screen (it is a view of the 로그 screen).
|
||||
|
||||
- **First-run wizard** (while `firstRun`): data folder (free space, warning under 200 GB), PC check,
|
||||
the model to download, Discord bot (token check with the
|
||||
@@ -884,7 +904,7 @@ are on the 앱 screen) and no separate 한 일 screen (it is a view of the 로
|
||||
Strata's last line, and stop in the header; the header with start, restart and stop is rebuilt
|
||||
only when the state changes, so a click is never lost to the refresh. Every screen change starts
|
||||
a new generation: an answer that arrives after the user moved to another screen is dropped.
|
||||
- **Settings screens** (봇, 채널·호출, 채팅 채널, 대화 기억, 모델, 기능, 백그라운드 수집, 예약, PC 제어, 게임 모드, 앱): built from `settings_fields`; only
|
||||
- **Settings screens** (봇, 음성 채널, 호출, 여럿이 말할 때, 채팅 채널, 대화 기억, 받아 적기·말하기, 대화 모델, 기능 켜기·MCP, 백그라운드 수집, 예약, PC 제어·화면 보기, 게임 모드, 앱): built from `settings_fields`, each field on one screen: 음성 채널 has the server, the auto-join channel, the speakers, bots, the join notice and the recording switch; 호출 the voice and chat wake words and `call.*`; 여럿이 말할 때 `response.*`; 받아 적기·말하기 `voice.stt.*` and the speech speed; 대화 모델 `llm.*`; the default location is on 기능 켜기·MCP (weather and search use it); only
|
||||
changed fields are sent; the server and voice channel are pickers (choosing another server clears
|
||||
the voice channel and lists only that server's channels); the chat channels are a table
|
||||
with their switches; the bot screen also changes the token; the 기능 screen has the feature
|
||||
|
||||
@@ -38,7 +38,7 @@ pub struct CallSettings {
|
||||
impl Default for CallSettings {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
wake_words: vec!["자비스".into()],
|
||||
wake_words: crate::settings::default_wake_words(),
|
||||
similarity: 0.2,
|
||||
mode: CallMode::WakeWord,
|
||||
designated: HashSet::new(),
|
||||
@@ -70,9 +70,9 @@ pub fn normalise(text: &str) -> Vec<char> {
|
||||
text.nfkd().flat_map(char::to_lowercase).filter(|c| c.is_alphanumeric()).collect()
|
||||
}
|
||||
|
||||
/// The stretch `[start, end)` of `text` closest to `word`, if within `max_edits` edits
|
||||
/// (approximate substring matching: the match may start and end anywhere in `text`).
|
||||
fn find(text: &[char], word: &[char], max_edits: usize) -> Option<(usize, usize)> {
|
||||
/// The stretch `[start, end)` of `text` closest to `word` and its edits, if within `max_edits`
|
||||
/// edits (approximate substring matching: the match may start and end anywhere in `text`).
|
||||
fn find(text: &[char], word: &[char], max_edits: usize) -> Option<(usize, usize, usize)> {
|
||||
if word.is_empty() || text.is_empty() {
|
||||
return None;
|
||||
}
|
||||
@@ -104,7 +104,7 @@ fn find(text: &[char], word: &[char], max_edits: usize) -> Option<(usize, usize)
|
||||
(1..=n)
|
||||
.filter(|&j| d[m][j] <= max_edits)
|
||||
.min_by_key(|&j| (d[m][j], j))
|
||||
.map(|j| (start[m][j], j))
|
||||
.map(|j| (start[m][j], j, d[m][j]))
|
||||
}
|
||||
|
||||
impl CallSettings {
|
||||
@@ -114,13 +114,15 @@ impl CallSettings {
|
||||
|
||||
/// The text with the first heard wake word removed, if one is heard.
|
||||
fn without_wake_word(&self, text: &[char]) -> Option<Vec<char>> {
|
||||
// The earliest heard wake word is the one used to address Jarvis.
|
||||
let (s, e) = self
|
||||
// The earliest heard wake word is the one used to address Jarvis; where several start
|
||||
// there, the closest, then the longest ("비서야" over "비서"), so no part of the call is
|
||||
// left over and a near match does not eat into what follows.
|
||||
let (s, e, _) = self
|
||||
.wake_words
|
||||
.iter()
|
||||
.map(|w| normalise(w))
|
||||
.filter_map(|word| find(text, &word, self.allowed_edits(&word)))
|
||||
.min()?;
|
||||
.min_by_key(|&(s, e, edits)| (s, edits, std::cmp::Reverse(e)))?;
|
||||
Some(text[..s].iter().chain(&text[e..]).copied().collect())
|
||||
}
|
||||
|
||||
@@ -169,8 +171,9 @@ mod tests {
|
||||
const A: u64 = 1;
|
||||
const B: u64 = 2;
|
||||
|
||||
/// The mechanism tests use one wake word of their own, whatever the default is.
|
||||
fn s() -> CallSettings {
|
||||
CallSettings::default()
|
||||
CallSettings { wake_words: vec!["자비스".into()], ..CallSettings::default() }
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -264,4 +267,25 @@ mod tests {
|
||||
assert_eq!(c.judge(A, "javis 날씨"), Call::Request { direct: true });
|
||||
assert_eq!(c.judge(A, "javis stop"), Call::Stop);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_longest_wake_word_where_the_call_starts_is_the_one_removed() {
|
||||
let c = CallSettings { wake_words: vec!["비서".into(), "비서야".into(), "김비서".into()], ..s() };
|
||||
assert!(c.wake_word_only("비서야,"), "the call form is the whole wake word, not 비서 and a 야 left over");
|
||||
assert!(c.wake_word_only("김비서?"));
|
||||
assert_eq!(c.judge(A, "비서야 멈춰"), Call::Stop);
|
||||
assert_eq!(c.judge(A, "김비서 그만해"), Call::Stop);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn by_default_jarvis_is_called_like_a_secretary() {
|
||||
let c = CallSettings::default();
|
||||
for call in ["비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"] {
|
||||
assert!(c.wake_word_only(&format!("{call},")), "{call}");
|
||||
assert_eq!(c.judge(A, &format!("{call} 오늘 날씨 어때?")), Call::Request { direct: true }, "{call}");
|
||||
assert_eq!(c.judge(A, &format!("{call} 멈춰")), Call::Stop, "{call}");
|
||||
}
|
||||
assert_eq!(c.judge(A, "오늘 날씨 어때?"), Call::Ignore);
|
||||
assert_eq!(c.judge(A, "자비스 날씨"), Call::Ignore, "the old name no longer calls");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -502,6 +502,11 @@ pub async fn run<V: Voice>(mut conv: Conversation<V>, mut events: mpsc::Unbounde
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The tests call Jarvis by one wake word of their own, whatever the default is.
|
||||
fn test_calls() -> CallSettings {
|
||||
CallSettings { wake_words: vec!["자비스".into()], ..CallSettings::default() }
|
||||
}
|
||||
use crate::discord::{Heard, HeardFinal};
|
||||
use crate::playback::PlaybackEnd;
|
||||
use crate::policy::PolicyKind;
|
||||
@@ -588,7 +593,7 @@ mod tests {
|
||||
|
||||
fn conv(kind: PolicyKind, mode: CallMode) -> (Conversation<Fake>, Fake, Instant) {
|
||||
let f = Fake::default();
|
||||
let calls = CallSettings { mode, earcon: false, ..CallSettings::default() };
|
||||
let calls = CallSettings { mode, earcon: false, ..test_calls() };
|
||||
let policy = PolicySettings { kind, collect: Duration::ZERO, ..PolicySettings::default() };
|
||||
*f.people.borrow_mut() = vec![A, B];
|
||||
(Conversation::new(f.clone(), calls, policy), f, Instant::now())
|
||||
@@ -596,7 +601,7 @@ mod tests {
|
||||
fn conv_earcon(kind: PolicyKind) -> (Conversation<Fake>, Fake, Instant) {
|
||||
let f = Fake::default();
|
||||
let policy = PolicySettings { kind, collect: Duration::ZERO, ..PolicySettings::default() };
|
||||
(Conversation::new(f.clone(), CallSettings::default(), policy), f, Instant::now())
|
||||
(Conversation::new(f.clone(), test_calls(), policy), f, Instant::now())
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -624,7 +629,7 @@ mod tests {
|
||||
// Nothing followed the next one: it is answered alone, still with one earcon.
|
||||
c.on_event(&end("s1", false), ms(t, 5000));
|
||||
c.on_event(&fin_utt(A, 3, "자비스"), ms(t, 6000));
|
||||
c.tick(ms(t, 6000) + CallSettings::default().follow_up);
|
||||
c.tick(ms(t, 6000) + test_calls().follow_up);
|
||||
assert_eq!(f.take(), vec![Did::Earcon, Did::Answer(A, "자비스".into())]);
|
||||
}
|
||||
|
||||
@@ -708,7 +713,7 @@ mod tests {
|
||||
c.on_event(&start(A, 1), ms(t, 0));
|
||||
c.on_event(&start(A, 2), ms(t, 600)); // the final of utterance 1 comes after 2 started
|
||||
c.on_event(&fin_utt(A, 1, "자비스"), ms(t, 900));
|
||||
c.tick(ms(t, 900) + CallSettings::default().follow_up + Duration::from_millis(1));
|
||||
c.tick(ms(t, 900) + test_calls().follow_up + Duration::from_millis(1));
|
||||
assert!(f.take().is_empty());
|
||||
c.on_event(&fin_utt(A, 2, "멈춰"), ms(t, 3000));
|
||||
assert_eq!(f.take(), vec![Did::StopAll], "joined, it is a stop");
|
||||
@@ -719,7 +724,7 @@ mod tests {
|
||||
let (mut c, f, t) = conv(PolicyKind::First, CallMode::WakeWord);
|
||||
c.on_event(&start(A, 1), ms(t, 0));
|
||||
c.on_event(&fin_utt(A, 1, "자비스"), ms(t, 900));
|
||||
let wait = CallSettings::default().follow_up;
|
||||
let wait = test_calls().follow_up;
|
||||
assert_eq!(c.next_deadline(), Some(ms(t, 900) + wait));
|
||||
c.tick(ms(t, 900) + wait - Duration::from_millis(1));
|
||||
assert!(f.take().is_empty());
|
||||
@@ -748,7 +753,7 @@ mod tests {
|
||||
assert!(f.take().is_empty(), "no wake word: not a call");
|
||||
c.on_event(&start(A, 2), ms(t, 62_000));
|
||||
c.on_event(&fin_utt(A, 2, "자비스"), ms(t, 63_000));
|
||||
let wait = CallSettings::default().follow_up;
|
||||
let wait = test_calls().follow_up;
|
||||
assert_eq!(c.next_deadline(), Some(ms(t, 63_000) + wait));
|
||||
c.tick(ms(t, 63_000) + wait);
|
||||
assert_eq!(f.take(), vec![Did::Answer(A, "자비스".into())]);
|
||||
@@ -759,7 +764,7 @@ mod tests {
|
||||
let (mut c, f, t) = conv(PolicyKind::First, CallMode::WakeWord);
|
||||
c.on_event(&fin_utt(A, 1, "자비스"), ms(t, 900));
|
||||
c.on_event(&CoreEvent::Stopped, ms(t, 1000));
|
||||
c.tick(ms(t, 900) + CallSettings::default().follow_up);
|
||||
c.tick(ms(t, 900) + test_calls().follow_up);
|
||||
assert!(f.take().is_empty());
|
||||
assert_eq!(c.next_deadline(), None);
|
||||
}
|
||||
@@ -826,7 +831,7 @@ mod tests {
|
||||
let f = Fake::default();
|
||||
*f.people.borrow_mut() = vec![A, B, 33];
|
||||
let calls = CallSettings { mode: CallMode::Addressed, earcon: false, designated: [A].into(), designated_only: true,
|
||||
..CallSettings::default() };
|
||||
..test_calls() };
|
||||
let mut c = Conversation::new(f.clone(), calls, PolicySettings { kind: PolicyKind::InOrder, ..PolicySettings::default() });
|
||||
c.on_event(&fin(A, "불 좀 꺼 줄래", 100), ms(t, 0));
|
||||
assert!(matches!(f.take()[..], [Did::Addressed(_, A)]));
|
||||
@@ -930,7 +935,7 @@ mod tests {
|
||||
(CH_ALL, ChatChannelCall { everything: true, send: true }),
|
||||
(CH_MUTE, ChatChannelCall { everything: true, send: false }),
|
||||
].into() };
|
||||
let voice = CallSettings { earcon: false, ..CallSettings::default() };
|
||||
let voice = CallSettings { earcon: false, ..test_calls() };
|
||||
let c = Conversation::new(f.clone(), voice, PolicySettings::default()).with_chat(chat);
|
||||
(c, f, Instant::now())
|
||||
}
|
||||
@@ -942,7 +947,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn a_chat_call_is_answered_as_a_reply_to_its_message() {
|
||||
let (mut c, f, t) = chat_conv(CallSettings::default());
|
||||
let (mut c, f, t) = chat_conv(test_calls());
|
||||
c.on_event(&chat(CH_WAKE, 5, A, false, "자비스 내일 일정 알려 줘"), t);
|
||||
let Did::ChatAnswer(id, user, text) = f.take().remove(0) else { panic!() };
|
||||
assert_eq!((user, text.as_str()), (A, "자비스 내일 일정 알려 줘"));
|
||||
@@ -956,7 +961,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn chat_answers_follow_each_channels_switch_and_never_answer_bots() {
|
||||
let (mut c, f, t) = chat_conv(CallSettings::default());
|
||||
let (mut c, f, t) = chat_conv(test_calls());
|
||||
c.on_event(&chat(CH_WAKE, 1, A, false, "오늘 뭐 먹지"), t);
|
||||
assert!(f.take().is_empty(), "wake word only");
|
||||
c.on_event(&chat(CH_ALL, 2, A, false, "오늘 뭐 먹지"), t);
|
||||
@@ -974,7 +979,7 @@ mod tests {
|
||||
#[test]
|
||||
fn chat_uses_its_own_wake_words_and_the_designated_people() {
|
||||
let calls = CallSettings { wake_words: vec!["봇".into()], designated: [A].into(), designated_only: true,
|
||||
..CallSettings::default() };
|
||||
..test_calls() };
|
||||
let (mut c, f, t) = chat_conv(calls);
|
||||
c.on_event(&chat(CH_WAKE, 1, A, false, "봇 안녕"), t);
|
||||
assert_eq!(f.take().len(), 1);
|
||||
@@ -986,7 +991,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn a_failed_chat_answer_posts_nothing_and_chat_leaves_voice_alone() {
|
||||
let (mut c, f, t) = chat_conv(CallSettings::default());
|
||||
let (mut c, f, t) = chat_conv(test_calls());
|
||||
c.on_event(&fin(A, "자비스 날씨", 100), t);
|
||||
c.on_event(&chat(CH_ALL, 9, B, false, "안녕"), t);
|
||||
let did = f.take();
|
||||
@@ -999,7 +1004,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn a_chat_answer_after_its_keep_time_is_not_posted() {
|
||||
let (mut c, f, t) = chat_conv(CallSettings::default());
|
||||
let (mut c, f, t) = chat_conv(test_calls());
|
||||
c.on_event(&chat(CH_WAKE, 5, A, false, "자비스 안녕"), t);
|
||||
let Did::ChatAnswer(id, ..) = f.take().remove(0) else { panic!() };
|
||||
c.on_event(&CoreEvent::ReplyText { id: id.clone(), text: "안녕하세요.".into(), failed: false }, t + CHAT_ANSWER_KEPT);
|
||||
|
||||
@@ -24,7 +24,7 @@ Settings (`settings.json`, section `call`; wake words stay in `voice.wakeWords`)
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `voice.wakeWords` | `["자비스"]` | Wake words for voice (also the STT hint). |
|
||||
| `voice.wakeWords` | `["비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"]` | Wake words for voice (also the STT hint). The default calls Jarvis like a secretary, by the word and its usual call forms (user request, 2026-11-19; before it was `["자비스"]`). |
|
||||
| `call.chatWakeWords` | `null` | Wake words for chat; `null` uses `voice.wakeWords`. |
|
||||
| `call.similarity` | `0.2` | How far a heard word may be from a wake word, as a share of the wake word's normalised letters (0-0.5). `0` needs an exact match. |
|
||||
| `call.mode` | `"wakeWord"` | `wakeWord`: answer only when called by a wake word. `addressed`: also answer what is judged to be said to Jarvis. `everything`: answer every utterance. |
|
||||
@@ -42,6 +42,9 @@ away from "자비스" (of six) while "서비스" is two away.
|
||||
A wake word is heard when some stretch of the normalised text is at most
|
||||
`floor(similarity × letters of the wake word)` edits (insert, delete, substitute) from it. With the
|
||||
default 0.2, "자비스" (six letters) allows one edit. A wake word anywhere in the utterance counts.
|
||||
The earliest heard wake word is the call; where several are heard starting there, the one with the
|
||||
fewest edits, then the longest, so a call form ("비서야") is removed whole and leaves nothing over
|
||||
("비서야 멈춰" is a stop), and a near match of a longer word does not eat into what follows.
|
||||
|
||||
Decision for one utterance (`judge(speaker, text)`), in order:
|
||||
|
||||
@@ -208,7 +211,7 @@ each to the chat model, the heard text fenced as data. Eval: `evals/test_voice_j
|
||||
## Tests
|
||||
|
||||
`cargo test` covers call detection (exact and near wake words in Hangul and Latin, a different word
|
||||
two letters away, wake word anywhere, stop with and without a wake word, designated combinations,
|
||||
two letters away, wake word anywhere, the longest call form where several start (the default secretary call forms), stop with and without a wake word, designated combinations,
|
||||
modes) and every policy rule above with fake times (settling, ordering by end time, busy, queue limit,
|
||||
interruption and `response.interrupt` off, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a
|
||||
fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed alone
|
||||
|
||||
@@ -856,6 +856,68 @@ fn report_voice_state(inner: &Inner) {
|
||||
debug_log!("voice", "reported voice {channel} with {n} members to the worker");
|
||||
}
|
||||
|
||||
/// One `screen_action` against the real screen.
|
||||
async fn screen_act(inner: &Arc<Inner>, req: crate::worker_client::ScreenAction) -> Result<serde_json::Value, String> {
|
||||
let settings = inner.settings.clone();
|
||||
tokio::task::spawn_blocking(move || match crate::screen::system_screen() {
|
||||
Some(screen) => crate::screen::act(&settings, screen.as_ref(), &req),
|
||||
None => Err("이 PC에서는 화면을 볼 수 없습니다".into()),
|
||||
}).await.unwrap_or_else(|_| Err("화면을 캡처하지 못했습니다".into()))
|
||||
}
|
||||
|
||||
/// How long a voice request waits for a screen share to be popped out, and how often it looks.
|
||||
const SHARE_WAIT: Duration = Duration::from_secs(30);
|
||||
const SHARE_POLL: Duration = Duration::from_secs(1);
|
||||
|
||||
/// Jarvis's voice channel's name and the people sharing their screen there (ID, display name);
|
||||
/// None outside a voice channel.
|
||||
fn share_state(inner: &Inner) -> Option<(Option<String>, Vec<(u64, String)>)> {
|
||||
let (guild, channel) = inner.session.lock().unwrap().as_ref().map(|s| (s.guild_id, s.channel_id))?;
|
||||
// IDs first, then names: the cache's guild is not held while `name_of` reads it again.
|
||||
let (name, users): (Option<String>, Vec<u64>) = inner.cache.get()
|
||||
.and_then(|c| c.guild(GuildId::new(guild)).map(|g| (
|
||||
g.channels.get(&ChannelId::new(channel)).map(|c| c.name.clone()),
|
||||
g.voice_states.values()
|
||||
.filter(|v| v.channel_id.map(|c| c.get()) == Some(channel) && v.self_stream == Some(true))
|
||||
.map(|v| v.user_id.get())
|
||||
.collect())))
|
||||
.unwrap_or_default();
|
||||
let bot = inner.bot_id.get().copied();
|
||||
Some((name, users.into_iter().filter(|u| Some(*u) != bot).map(|u| (u, name_of(inner, u))).collect()))
|
||||
}
|
||||
|
||||
/// Someone's Discord screen share (admin.spec.md, "Screen viewing"): only while the person asked
|
||||
/// about shares their screen in Jarvis's voice channel, from the window Discord pops it out to. With
|
||||
/// no such window a voice request asks for it in the channel and waits `SHARE_WAIT`.
|
||||
async fn share_capture(inner: &Arc<Inner>, mut req: crate::worker_client::ScreenAction) -> Result<serde_json::Value, String> {
|
||||
// Who may see comes first: nothing is said or waited for for someone who may not.
|
||||
if let Some(why) = crate::screen::refusal(&inner.settings, &req.user) {
|
||||
return Err(why);
|
||||
}
|
||||
let (channel, streaming) = share_state(inner).ok_or("자비스가 음성 채널에 있을 때만 화면공유를 볼 수 있습니다")?;
|
||||
let name = crate::screen::share_streamer(&streaming, req.person.as_deref())?;
|
||||
req.voice_channel = channel;
|
||||
let first = screen_act(inner, req.clone()).await;
|
||||
if first.as_ref().err().map(String::as_str) != Some(crate::screen::NO_SHARE_WINDOW) {
|
||||
return first;
|
||||
}
|
||||
let prompt = crate::screen::share_prompt(name.as_deref());
|
||||
if req.via.as_deref() != Some("voice") {
|
||||
return Err(format!("{} 그 뒤 다시 물어봐 주세요", prompt.trim_end_matches('.')));
|
||||
}
|
||||
debug_log!("conversation", "screen share {}: no popped-out window, asking and waiting", req.id);
|
||||
with_playback(inner, |p| p.say(&prompt, false));
|
||||
let until = tokio::time::Instant::now() + SHARE_WAIT;
|
||||
while tokio::time::Instant::now() < until {
|
||||
tokio::time::sleep(SHARE_POLL).await;
|
||||
let r = screen_act(inner, req.clone()).await;
|
||||
if r.as_ref().err().map(String::as_str) != Some(crate::screen::NO_SHARE_WINDOW) {
|
||||
return r;
|
||||
}
|
||||
}
|
||||
Err("화면공유 창이 분리되지 않아 볼 수 없습니다".into())
|
||||
}
|
||||
|
||||
/// The registered chat channels and their switches, for the brain's chat tool.
|
||||
fn report_chat_channels(inner: &Inner) {
|
||||
inner.worker.chat_channels(inner.bot_id.get().copied(), &inner.settings.chat_channels);
|
||||
@@ -1003,12 +1065,11 @@ async fn worker_task(core: std::sync::Weak<Inner>, mut rx: mpsc::UnboundedReceiv
|
||||
let inner = inner.clone();
|
||||
tokio::spawn(async move {
|
||||
let started = std::time::Instant::now();
|
||||
let settings = inner.settings.clone();
|
||||
let req = a.clone();
|
||||
let result = tokio::task::spawn_blocking(move || match crate::screen::system_screen() {
|
||||
Some(screen) => crate::screen::act(&settings, screen.as_ref(), &req),
|
||||
None => Err("이 PC에서는 화면을 볼 수 없습니다".into()),
|
||||
}).await.unwrap_or_else(|_| Err("화면을 캡처하지 못했습니다".into()));
|
||||
let result = if a.action == "capture" && a.target.as_deref() == Some("share") {
|
||||
share_capture(&inner, a.clone()).await
|
||||
} else {
|
||||
screen_act(&inner, a.clone()).await
|
||||
};
|
||||
// The kind of target, size and time only: never a title or a picture.
|
||||
let target = crate::screen::target_of(&a).map(|(t, _)| crate::screen::activity_target(&t))
|
||||
.unwrap_or_else(|_| "?".into());
|
||||
|
||||
@@ -96,6 +96,8 @@ pub enum Target {
|
||||
Screen,
|
||||
Monitor(u32),
|
||||
Window(String),
|
||||
/// Someone's Discord screen share, in the window Discord pops it out to.
|
||||
Share,
|
||||
}
|
||||
|
||||
/// The activity log's target: never a title.
|
||||
@@ -104,6 +106,7 @@ pub fn activity_target(t: &Target) -> String {
|
||||
Target::Screen => "전체 화면".into(),
|
||||
Target::Monitor(n) => format!("모니터 {n}"),
|
||||
Target::Window(_) => "창".into(),
|
||||
Target::Share => "화면공유".into(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -120,6 +123,7 @@ pub fn target_of(a: &ScreenAction) -> Result<(Target, Detail), String> {
|
||||
let w = a.window.as_deref().map(str::trim).filter(|w| !w.is_empty()).ok_or("창 이름이 필요합니다")?;
|
||||
Target::Window(w.chars().take(200).collect())
|
||||
}
|
||||
Some("share") => Target::Share,
|
||||
_ => Target::Screen,
|
||||
};
|
||||
Ok((target, detail))
|
||||
@@ -169,6 +173,55 @@ pub fn pick_window<'a>(s: &'a Settings, all: &'a [Window], text: &str) -> Option
|
||||
}
|
||||
|
||||
/// Whether nothing shows (protected video, exclusive full screen): every pixel near black.
|
||||
/// Discord's programs (stable, PTB, Canary).
|
||||
const DISCORD: [&str; 3] = ["discord.exe", "discordptb.exe", "discordcanary.exe"];
|
||||
|
||||
/// What a share capture gives while Discord has no popped-out screen share window: the core then
|
||||
/// asks for one and waits (admin.spec.md, "Screen viewing").
|
||||
pub const NO_SHARE_WINDOW: &str = "분리된 화면공유 창이 없습니다";
|
||||
|
||||
/// The window Discord pops a screen share out to: a Discord window that is not its main one (whose
|
||||
/// title ends with "Discord"). Its title is the voice channel's name, so the one naming Jarvis's
|
||||
/// voice channel goes first, else the one nearest the front.
|
||||
pub fn share_window<'a>(s: &'a Settings, all: &'a [Window], channel: Option<&str>) -> Option<&'a Window> {
|
||||
let shares: Vec<&Window> = listable(s, all)
|
||||
.filter(|w| DISCORD.contains(&w.program.as_str()) && !w.title.trim().to_lowercase().ends_with("discord"))
|
||||
.collect();
|
||||
let channel = channel.map(|c| c.trim().to_lowercase()).filter(|c| !c.is_empty());
|
||||
channel.and_then(|c| shares.iter().find(|w| w.title.to_lowercase().contains(&c)).copied())
|
||||
.or_else(|| shares.first().copied())
|
||||
}
|
||||
|
||||
/// Letters and digits in lower case, so a spoken name and a display name compare.
|
||||
fn name_key(name: &str) -> String {
|
||||
name.chars().filter(|c| c.is_alphanumeric()).flat_map(char::to_lowercase).collect()
|
||||
}
|
||||
|
||||
/// Whose share the popped-out window should show, from the people sharing their screen in Jarvis's
|
||||
/// voice channel (ID, display name): the person asked about (their name inside what was said, or
|
||||
/// what was said inside their name), or with no name anyone sharing (`None` when several do).
|
||||
pub fn share_streamer(streaming: &[(u64, String)], person: Option<&str>) -> Result<Option<String>, String> {
|
||||
let person = person.map(str::trim).filter(|p| !name_key(p).is_empty());
|
||||
let Some(person) = person else {
|
||||
return match streaming {
|
||||
[] => Err("지금 음성 채널에서 화면공유 중인 사람이 없습니다".into()),
|
||||
[(_, name)] => Ok(Some(name.clone())),
|
||||
_ => Ok(None),
|
||||
};
|
||||
};
|
||||
let key = name_key(person);
|
||||
streaming.iter()
|
||||
.find(|(_, name)| { let n = name_key(name); !n.is_empty() && (key.contains(&n) || n.contains(&key)) })
|
||||
.map(|(_, name)| Some(name.clone()))
|
||||
.ok_or_else(|| format!("{}님은 지금 화면공유 중이 아닙니다", person.chars().take(40).collect::<String>()))
|
||||
}
|
||||
|
||||
/// What Jarvis says in the voice channel while it waits for the share to be popped out.
|
||||
pub fn share_prompt(name: Option<&str>) -> String {
|
||||
let whose = name.map(|n| format!("{n}님 화면공유를")).unwrap_or_else(|| "화면공유를".into());
|
||||
format!("디스코드에서 {whose} 시청하고 창 분리 버튼을 눌러 주세요.")
|
||||
}
|
||||
|
||||
pub fn all_black(img: &Image) -> bool {
|
||||
img.bgra.chunks_exact(4).all(|p| p[0] <= 8 && p[1] <= 8 && p[2] <= 8)
|
||||
}
|
||||
@@ -330,6 +383,16 @@ pub fn act(s: &Settings, screen: &dyn Screen, a: &ScreenAction) -> Result<Value,
|
||||
let black = all_black(&img);
|
||||
(img, black, format!("창 {}", w.program.strip_suffix(".exe").unwrap_or(&w.program)))
|
||||
}
|
||||
Target::Share => {
|
||||
let all = screen.windows()?;
|
||||
let w = share_window(s, &all, a.voice_channel.as_deref()).ok_or(NO_SHARE_WINDOW)?;
|
||||
if w.minimised {
|
||||
return Err("화면공유 창이 최소화되어 있어 볼 수 없습니다. 창을 다시 열어 주세요".into());
|
||||
}
|
||||
let img = screen.capture_window(w)?;
|
||||
let black = all_black(&img);
|
||||
(img, black, activity_target(&target))
|
||||
}
|
||||
_ => {
|
||||
let ms = screen.monitors()?;
|
||||
let chosen: Vec<Monitor> = match target {
|
||||
@@ -813,6 +876,73 @@ mod tests {
|
||||
assert!(v.get("images").is_none());
|
||||
}
|
||||
|
||||
fn with_share(f: &mut Fake, popout: Window) {
|
||||
f.windows.insert(1, win("discord.exe", "#일반 | 우리 서버 - Discord", Rect { x: 0, y: 0, w: 1200, h: 800 }));
|
||||
f.windows.insert(0, popout);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_screen_share_is_seen_in_discords_popped_out_window_never_its_main_one() {
|
||||
let s = settings();
|
||||
let mut f = fake();
|
||||
let mut a = action("capture", OWNER, json!({"target": "share", "person": "철수"}));
|
||||
a.voice_channel = Some("게임방".into());
|
||||
// Only the main window: nothing popped out yet.
|
||||
f.windows.insert(1, win("discord.exe", "#일반 | 우리 서버 - Discord", Rect { x: 0, y: 0, w: 1200, h: 800 }));
|
||||
assert_eq!(act(&s, &f, &a).unwrap_err(), NO_SHARE_WINDOW);
|
||||
assert!(f.captured.borrow().is_empty());
|
||||
|
||||
let mut f = fake();
|
||||
with_share(&mut f, win("discord.exe", "게임방", Rect { x: 0, y: 0, w: 960, h: 540 }));
|
||||
let v = act(&s, &f, &a).unwrap();
|
||||
assert_eq!(*f.captured.borrow(), vec!["window discord.exe"]);
|
||||
assert_eq!(v["item"]["what"], "화면공유");
|
||||
assert_eq!(activity_target(&target_of(&a).unwrap().0), "화면공유");
|
||||
|
||||
// A popped-out window that is minimised is asked to be opened again.
|
||||
let mut f = fake();
|
||||
with_share(&mut f, Window { minimised: true, ..win("discord.exe", "게임방", Rect { x: 0, y: 0, w: 960, h: 540 }) });
|
||||
assert!(act(&s, &f, &a).unwrap_err().contains("최소화"));
|
||||
|
||||
// Discord on the exclusion list is never seen, popped out or not.
|
||||
let mut s = settings();
|
||||
s.pc_control.screen_exclude.push("Discord".into());
|
||||
let mut f = fake();
|
||||
with_share(&mut f, win("discord.exe", "게임방", Rect { x: 0, y: 0, w: 960, h: 540 }));
|
||||
assert_eq!(act(&s, &f, &a).unwrap_err(), NO_SHARE_WINDOW);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_popped_out_window_naming_jarvis_voice_channel_goes_first() {
|
||||
let s = settings();
|
||||
let all = vec![
|
||||
win("discordptb.exe", "다른 방", Rect { x: 0, y: 0, w: 10, h: 10 }),
|
||||
win("discordptb.exe", "게임방", Rect { x: 0, y: 0, w: 10, h: 10 }),
|
||||
win("discordptb.exe", "Discord", Rect { x: 0, y: 0, w: 10, h: 10 }),
|
||||
];
|
||||
assert_eq!(share_window(&s, &all, Some("게임방")).unwrap().title, "게임방");
|
||||
assert_eq!(share_window(&s, &all, None).unwrap().title, "다른 방", "else the one nearest the front");
|
||||
assert!(share_window(&s, &all[2..], Some("게임방")).is_none(), "the main window is never the share");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_share_is_seen_only_while_the_person_asked_about_shares_their_screen() {
|
||||
let streaming = vec![(1, "철수".to_owned()), (2, "Younghee".to_owned())];
|
||||
assert_eq!(share_streamer(&streaming, Some("철수님")), Ok(Some("철수".to_owned())), "a spoken suffix still matches");
|
||||
assert_eq!(share_streamer(&streaming, Some("younghee")), Ok(Some("Younghee".to_owned())));
|
||||
assert_eq!(share_streamer(&streaming, Some("민수")).unwrap_err(), "민수님은 지금 화면공유 중이 아닙니다");
|
||||
assert_eq!(share_streamer(&streaming, None), Ok(None), "no name: whatever the window shows");
|
||||
assert_eq!(share_streamer(&streaming[..1], Some(" ")), Ok(Some("철수".to_owned())));
|
||||
assert!(share_streamer(&[], None).unwrap_err().contains("화면공유 중인 사람이 없습니다"));
|
||||
assert!(share_streamer(&[], Some("철수")).unwrap_err().contains("철수님은"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_share_prompt_names_the_person_when_it_is_known() {
|
||||
assert!(share_prompt(Some("철수")).contains("철수님 화면공유"));
|
||||
assert!(share_prompt(None).contains("화면공유를"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scaling_averages_the_area() {
|
||||
let mut img = solid(4, 2, 0);
|
||||
|
||||
@@ -254,6 +254,12 @@ impl std::error::Error for SettingsError {}
|
||||
pub const DEFAULT_JOIN_NOTICE: &str =
|
||||
"자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다.";
|
||||
|
||||
/// The default wake words: Jarvis is called like a secretary, by the word and its usual call forms
|
||||
/// (the longest one heard is the call, `calls.rs`).
|
||||
pub fn default_wake_words() -> Vec<String> {
|
||||
["비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"].map(String::from).to_vec()
|
||||
}
|
||||
|
||||
impl Default for Settings {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
@@ -268,7 +274,7 @@ impl Default for Settings {
|
||||
listen_to_bots: false,
|
||||
record: false,
|
||||
join_notice: DEFAULT_JOIN_NOTICE.into(),
|
||||
wake_words: vec!["자비스".into()],
|
||||
wake_words: default_wake_words(),
|
||||
stt: Map::new(),
|
||||
tts_speed: 1.5,
|
||||
},
|
||||
|
||||
@@ -15,10 +15,13 @@ use crate::settings::{parse_settings, DEFAULT_JOIN_NOTICE};
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub enum Screen {
|
||||
Bot,
|
||||
Channels,
|
||||
VoiceChannel,
|
||||
Call,
|
||||
Response,
|
||||
Chat,
|
||||
Memory,
|
||||
Model,
|
||||
Voice,
|
||||
Llm,
|
||||
Features,
|
||||
Collect,
|
||||
Game,
|
||||
@@ -78,33 +81,33 @@ pub fn fields() -> Vec<Field> {
|
||||
vec![
|
||||
f("discord.commandName", Bot, Text { max: 32, non_empty: true }, json!("자비스"), "슬래시 명령 이름"),
|
||||
f("discord.ephemeralReplies", Bot, Bool, json!(true), "명령 응답은 부른 사람만 보기"),
|
||||
f("discord.guildId", Channels, Id, Value::Null, "서버"),
|
||||
f("discord.voiceChannelId", Channels, Id, Value::Null, "자동으로 들어갈 음성 채널"),
|
||||
f("voice.maxSpeakers", Channels, Int { min: 1, max: 10 }, json!(10), "동시에 받아 적을 최대 인원"),
|
||||
f("voice.listenToBots", Channels, Bool, json!(false), "다른 봇의 말도 받아 적기"),
|
||||
f("voice.wakeWords", Channels, TextList { max_items: 20, nullable: false }, json!(["자비스"]), "호출어"),
|
||||
f("call.mode", Channels, Choice { values: &["wakeWord", "addressed", "everything"] }, json!("wakeWord"),
|
||||
f("discord.guildId", VoiceChannel, Id, Value::Null, "서버"),
|
||||
f("discord.voiceChannelId", VoiceChannel, Id, Value::Null, "자동으로 들어갈 음성 채널"),
|
||||
f("voice.maxSpeakers", VoiceChannel, Int { min: 1, max: 10 }, json!(10), "동시에 받아 적을 최대 인원"),
|
||||
f("voice.listenToBots", VoiceChannel, Bool, json!(false), "다른 봇의 말도 받아 적기"),
|
||||
f("voice.wakeWords", Call, TextList { max_items: 20, nullable: false }, json!(crate::settings::default_wake_words()), "호출어"),
|
||||
f("call.mode", Call, Choice { values: &["wakeWord", "addressed", "everything"] }, json!("wakeWord"),
|
||||
"호출 방식"),
|
||||
f("call.similarity", Channels, Number { min: 0.0, max: 0.5 }, json!(0.2), "비슷한 발음 허용 정도"),
|
||||
f("call.designated", Channels, IdList, json!([]), "지정한 사람"),
|
||||
f("call.designatedOnly", Channels, Bool, json!(false), "지정한 사람에게만 응답"),
|
||||
f("call.designatedNoWakeWord", Channels, Bool, json!(false), "지정한 사람은 호출어 없이 대답"),
|
||||
f("call.stopWords", Channels, TextList { max_items: 20, nullable: false },
|
||||
f("call.similarity", Call, Number { min: 0.0, max: 0.5 }, json!(0.2), "비슷한 발음 허용 정도"),
|
||||
f("call.designated", Call, IdList, json!([]), "지정한 사람"),
|
||||
f("call.designatedOnly", Call, Bool, json!(false), "지정한 사람에게만 응답"),
|
||||
f("call.designatedNoWakeWord", Call, Bool, json!(false), "지정한 사람은 호출어 없이 대답"),
|
||||
f("call.stopWords", Call, TextList { max_items: 20, nullable: false },
|
||||
json!(["멈춰", "그만", "정지", "stop"]), "정지 명령"),
|
||||
f("call.followUpMs", Channels, Int { min: 0, max: 5000 }, json!(1500), "호출어 뒤 이어 말하기 대기(ms)"),
|
||||
f("call.earcon", Channels, Bool, json!(true), "부르면 효과음"),
|
||||
f("response.policy", Channels, Choice { values: &["first", "inOrder", "latest", "important"] },
|
||||
f("call.followUpMs", Call, Int { min: 0, max: 5000 }, json!(1500), "호출어 뒤 이어 말하기 대기(ms)"),
|
||||
f("call.earcon", Call, Bool, json!(true), "부르면 효과음"),
|
||||
f("response.policy", Response, Choice { values: &["first", "inOrder", "latest", "important"] },
|
||||
json!("latest"), "동시 대화 응답 정책"),
|
||||
f("response.queueMax", Channels, Int { min: 1, max: 20 }, json!(5), "기다리는 요청 최대 개수"),
|
||||
f("response.collectMs", Channels, Int { min: 0, max: 3000 }, json!(700), "중요도 비교 대기(ms)"),
|
||||
f("response.settleMs", Channels, Int { min: 0, max: 3000 }, json!(800), "먼저 끝난 말 전사 대기(ms)"),
|
||||
f("response.priorities", Channels, IdNumberMap, json!({}), "사람별 우선순위"),
|
||||
f("response.interrupt", Channels, Bool, json!(true),
|
||||
f("response.queueMax", Response, Int { min: 1, max: 20 }, json!(5), "기다리는 요청 최대 개수"),
|
||||
f("response.collectMs", Response, Int { min: 0, max: 3000 }, json!(700), "중요도 비교 대기(ms)"),
|
||||
f("response.settleMs", Response, Int { min: 0, max: 3000 }, json!(800), "먼저 끝난 말 전사 대기(ms)"),
|
||||
f("response.priorities", Response, IdNumberMap, json!({}), "사람별 우선순위"),
|
||||
f("response.interrupt", Response, Bool, json!(true),
|
||||
"말하는 중에 새 요청이 오면 하던 말 멈추기(가장 최근 것, 중요한 것 먼저)"),
|
||||
f("voice.joinNotice", Channels, Text { max: 2000, non_empty: true }, json!(DEFAULT_JOIN_NOTICE),
|
||||
f("voice.joinNotice", VoiceChannel, Text { max: 2000, non_empty: true }, json!(DEFAULT_JOIN_NOTICE),
|
||||
"입장 시 전사 고지 문구"),
|
||||
f("chatChannels", Chat, ChatChannels, json!([]), "채팅 채널"),
|
||||
f("call.chatWakeWords", Chat, TextList { max_items: 20, nullable: true }, Value::Null,
|
||||
f("call.chatWakeWords", Call, TextList { max_items: 20, nullable: true }, Value::Null,
|
||||
"채팅 호출어(비우면 음성과 같음)"),
|
||||
// The worker's memory settings and their defaults (voice_worker.spec.md, "Memory settings").
|
||||
f("memory.chat", Memory, Bool, json!(true), "등록한 채팅 채널 참고"),
|
||||
@@ -116,21 +119,21 @@ pub fn fields() -> Vec<Field> {
|
||||
f("memory.recent_records", Memory, Int { min: 0, max: 200 }, json!(20), "최근 대화 원문 개수"),
|
||||
f("memory.matches", Memory, Int { min: 0, max: 20 }, json!(5), "과거 기록 가져올 개수"),
|
||||
f("memory.budget_tokens", Memory, Number { min: 0.0, max: 100000.0 }, json!(4000), "참고 분량 상한(토큰)"),
|
||||
f("voice.record", Memory, Bool, json!(false),
|
||||
f("voice.record", VoiceChannel, Bool, json!(false),
|
||||
"목소리 녹음 저장(사람 말과 자비스 말, 분석용, 7일 보관, 다음 시작부터)"),
|
||||
// The worker's STT settings and their defaults (voice_worker.spec.md, "STT").
|
||||
f("voice.stt.end_silence_ms", Model, Int { min: 100, max: 2000 }, json!(300), "말 끝 판정 시간(ms)"),
|
||||
f("voice.stt.partials", Model, Bool, json!(true), "중간 전사"),
|
||||
f("voice.stt.partial_interval_ms", Model, Int { min: 100, max: 5000 }, json!(400), "중간 전사 간격(ms)"),
|
||||
f("voice.stt.final_beam", Model, Int { min: 1, max: 5 }, json!(2), "최종 전사 정확도(빔)"),
|
||||
f("voice.stt.language", Model, LanguageCode, json!("ko"), "받아 적을 언어(비우면 자동)"),
|
||||
f("voice.stt.end_silence_ms", Voice, Int { min: 100, max: 2000 }, json!(300), "말 끝 판정 시간(ms)"),
|
||||
f("voice.stt.partials", Voice, Bool, json!(true), "중간 전사"),
|
||||
f("voice.stt.partial_interval_ms", Voice, Int { min: 100, max: 5000 }, json!(400), "중간 전사 간격(ms)"),
|
||||
f("voice.stt.final_beam", Voice, Int { min: 1, max: 5 }, json!(2), "최종 전사 정확도(빔)"),
|
||||
f("voice.stt.language", Voice, LanguageCode, json!("ko"), "받아 적을 언어(비우면 자동)"),
|
||||
// MeloTTS's Korean model has one voice (speaker "KR"), so only the speed is a choice.
|
||||
f("voice.tts.speed", Model, Number { min: 0.5, max: 2.0 }, json!(1.5), "말하기 속도(1.0 보통, 기본 1.5, 저장하면 바로 적용)"),
|
||||
f("voice.tts.speed", Voice, Number { min: 0.5, max: 2.0 }, json!(1.5), "말하기 속도(1.0 보통, 기본 1.5, 저장하면 바로 적용)"),
|
||||
// The LLM (admin.spec.md, "LLM"); `llm.url` (a server the app does not start) is file-only.
|
||||
f("llm.context", Model, Int { min: 4096, max: 262144 }, json!(32768), "대화 모델이 한 번에 기억하는 양(토큰)"),
|
||||
f("brain.location", Model, Text { max: 100, non_empty: false }, json!(crate::settings::DEFAULT_LOCATION),
|
||||
f("llm.context", Llm, Int { min: 4096, max: 262144 }, json!(32768), "대화 모델이 한 번에 기억하는 양(토큰)"),
|
||||
f("brain.location", Features, Text { max: 100, non_empty: false }, json!(crate::settings::DEFAULT_LOCATION),
|
||||
"기본 위치(날씨, 검색, 추천의 기준, 비우면 자동 감지)"),
|
||||
f("llm.port", Model, Int { min: 1024, max: 65535 }, json!(8080), "대화 모델 포트"),
|
||||
f("llm.port", Llm, Int { min: 1024, max: 65535 }, json!(8080), "대화 모델 포트"),
|
||||
// The brain's features (admin.spec.md, "Features and MCP servers").
|
||||
f("features.weather", Features, Bool, json!(true), "날씨"),
|
||||
f("features.webSearch", Features, Bool, json!(true), "웹 검색"),
|
||||
|
||||
@@ -172,16 +172,25 @@ pub struct ScreenAction {
|
||||
/// `capture` or `list`.
|
||||
pub action: String,
|
||||
pub user: String,
|
||||
/// capture: `screen`, `monitor` or `window`.
|
||||
/// `voice` or `chat`.
|
||||
#[serde(default)]
|
||||
pub via: Option<String>,
|
||||
/// capture: `screen`, `monitor`, `window` or `share`.
|
||||
#[serde(default)]
|
||||
pub target: Option<String>,
|
||||
#[serde(default)]
|
||||
pub monitor: Option<u32>,
|
||||
#[serde(default)]
|
||||
pub window: Option<String>,
|
||||
/// share: the name of the person whose screen share is asked about.
|
||||
#[serde(default)]
|
||||
pub person: Option<String>,
|
||||
/// `fast`, `normal` or `detailed`.
|
||||
#[serde(default)]
|
||||
pub detail: Option<String>,
|
||||
/// share: the name of Jarvis's voice channel, set by the core (the popped-out window's title).
|
||||
#[serde(skip)]
|
||||
pub voice_channel: Option<String>,
|
||||
}
|
||||
|
||||
/// The brain's pcControl tool asks the app for the person asking (voice_worker.spec.md, "PC control").
|
||||
|
||||
@@ -210,8 +210,8 @@ async fn conversation_round_trip_on_real_discord() {
|
||||
}
|
||||
std::fs::write(data.join("config").join("settings.json"), serde_json::to_string_pretty(&json!({
|
||||
"discord": {"guildId": guild.to_string(), "voiceChannelId": voice.to_string()},
|
||||
// The "person" here is a bot, so bots must be heard.
|
||||
"voice": {"listenToBots": true},
|
||||
// The "person" here is a bot, so bots must be heard; the recorded questions call "자비스".
|
||||
"voice": {"listenToBots": true, "wakeWords": ["자비스"]},
|
||||
"voiceWorker": {"python": env("JARVIS_VOICE_PYTHON"), "cwd": root.join("../src"), "args": args},
|
||||
"llm": {"url": env("JARVIS_LIVE_LLM_URL")},
|
||||
})).unwrap()).unwrap();
|
||||
|
||||
@@ -992,7 +992,7 @@ function updateCard() {
|
||||
async function checkUpdate() {
|
||||
try {
|
||||
const u = await api("update_check");
|
||||
if (u.latest) { $("#nav-app").textContent = "앱 (새 버전)"; toast(`새 버전 ${u.latest.version}이 있습니다. 앱 화면에서 설치할 수 있습니다.`); }
|
||||
if (u.latest) { $("#nav-app").textContent = "앱 (새 버전)"; setGroup($("#nav-app").closest(".group"), true); toast(`새 버전 ${u.latest.version}이 있습니다. 앱 화면에서 설치할 수 있습니다.`); }
|
||||
} catch (e) { /* offline: the 앱 screen shows why */ }
|
||||
}
|
||||
|
||||
@@ -1025,14 +1025,18 @@ async function show(name) {
|
||||
current = name;
|
||||
clearInterval(ticker);
|
||||
for (const b of document.querySelectorAll(".nav")) b.classList.toggle("on", b.dataset.screen === name);
|
||||
// The group of the screen shown opens (a new version's 앱 button is then seen too).
|
||||
const group = document.querySelector(`.nav[data-screen="${name}"]`)?.closest(".group");
|
||||
if (group) setGroup(group, true);
|
||||
const screen = $("#screen");
|
||||
try {
|
||||
if (name === "dashboard") { screen.replaceChildren(); await refresh(); ticker = setInterval(refresh, 1000); return; }
|
||||
if (name === "logs") { const l = logsScreen(); screen.replaceChildren(l.view); ticker = setInterval(l.tick, 1000); return; }
|
||||
if (name === "conversations") { const c = conversationScreen(); screen.replaceChildren(c.view); ticker = setInterval(c.tick, 500); return; }
|
||||
if (name === "notes") { screen.replaceChildren(notesScreen()); return; }
|
||||
const titles = { bot: "봇", channels: "채널·호출", chat: "채팅 채널", memory: "대화 기억", model: "모델", features: "기능",
|
||||
collect: "백그라운드 수집", game: "게임 모드", schedule: "예약", pcControl: "PC 제어", app: "앱" };
|
||||
const titles = { bot: "봇", voiceChannel: "음성 채널", call: "호출", response: "여럿이 말할 때", chat: "채팅 채널",
|
||||
memory: "대화 기억", voice: "받아 적기·말하기", llm: "대화 모델", features: "기능 켜기·MCP",
|
||||
collect: "백그라운드 수집", game: "게임 모드", schedule: "예약", pcControl: "PC 제어·화면 보기", app: "앱" };
|
||||
const update = name === "app" ? updateCard() : null;
|
||||
const extra = name === "bot" ? await tokenCard() : name === "features" ? h("div", {}, await kmaKeyCard(), await displayDriverCard())
|
||||
: name === "app" ? h("div", {}, update.view, h("div", { class: "hint", style: "margin:12px 0" }, "트레이로 최소화하면 창만 숨고 봇과 서비스는 계속 돕니다. 작업 표시줄 오른쪽의 Jarvis 아이콘을 누르면 창이 다시 열리고, 아이콘의 오른쪽 클릭 메뉴에서 완전 종료할 수 있습니다. 저장하면 바로 적용됩니다.")) : null;
|
||||
@@ -1046,6 +1050,23 @@ async function show(name) {
|
||||
} catch (e) { fail(e); }
|
||||
}
|
||||
|
||||
/** Open or close one menu group; which groups are open is kept for the next start. */
|
||||
function setGroup(g, open) {
|
||||
g.classList.toggle("open", open);
|
||||
g.querySelector(".group-head").setAttribute("aria-expanded", String(open));
|
||||
const kept = new Set(JSON.parse(localStorage.getItem("openGroups") || "[]"));
|
||||
open ? kept.add(g.dataset.group) : kept.delete(g.dataset.group);
|
||||
localStorage.setItem("openGroups", JSON.stringify([...kept]));
|
||||
}
|
||||
|
||||
function initGroups() {
|
||||
const kept = new Set(JSON.parse(localStorage.getItem("openGroups") || "[]"));
|
||||
for (const g of document.querySelectorAll(".group")) {
|
||||
if (kept.has(g.dataset.group)) setGroup(g, true);
|
||||
g.querySelector(".group-head").onclick = () => setGroup(g, !g.classList.contains("open"));
|
||||
}
|
||||
}
|
||||
|
||||
// ---- first-run wizard ---------------------------------------------------------------------------
|
||||
|
||||
const WIZ = ["데이터 폴더", "PC 점검", "모델", "디스코드 봇", "서버·채널", "호출어", "완료"];
|
||||
@@ -1213,6 +1234,7 @@ async function start() {
|
||||
$("#wizard").hidden = true;
|
||||
$("#main").hidden = false;
|
||||
for (const b of document.querySelectorAll(".nav")) b.onclick = () => show(b.dataset.screen);
|
||||
initGroups();
|
||||
show("dashboard");
|
||||
checkUpdate();
|
||||
}
|
||||
|
||||
@@ -13,19 +13,52 @@
|
||||
<div class="brand">Jarvis</div>
|
||||
<button data-screen="dashboard" class="nav">대시보드</button>
|
||||
<button data-screen="conversations" class="nav">대화</button>
|
||||
<button data-screen="bot" class="nav">봇</button>
|
||||
<button data-screen="channels" class="nav">채널·호출</button>
|
||||
<button data-screen="chat" class="nav">채팅 채널</button>
|
||||
<button data-screen="memory" class="nav">대화 기억</button>
|
||||
<button data-screen="model" class="nav">모델</button>
|
||||
<button data-screen="features" class="nav">기능</button>
|
||||
<button data-screen="collect" class="nav">백그라운드 수집</button>
|
||||
<button data-screen="schedule" class="nav">예약</button>
|
||||
<button data-screen="pcControl" class="nav">PC 제어</button>
|
||||
<button data-screen="notes" class="nav">사람별 지침</button>
|
||||
<button data-screen="game" class="nav">게임 모드</button>
|
||||
<button data-screen="logs" class="nav">로그</button>
|
||||
<button data-screen="app" class="nav" id="nav-app">앱</button>
|
||||
<div class="group" data-group="discord">
|
||||
<button class="group-head" aria-expanded="false">디스코드</button>
|
||||
<div class="group-items">
|
||||
<button data-screen="bot" class="nav">봇</button>
|
||||
<button data-screen="voiceChannel" class="nav">음성 채널</button>
|
||||
<button data-screen="chat" class="nav">채팅 채널</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="group" data-group="talk">
|
||||
<button class="group-head" aria-expanded="false">대화 방식</button>
|
||||
<div class="group-items">
|
||||
<button data-screen="call" class="nav">호출</button>
|
||||
<button data-screen="response" class="nav">여럿이 말할 때</button>
|
||||
<button data-screen="memory" class="nav">대화 기억</button>
|
||||
<button data-screen="notes" class="nav">사람별 지침</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="group" data-group="model">
|
||||
<button class="group-head" aria-expanded="false">음성·모델</button>
|
||||
<div class="group-items">
|
||||
<button data-screen="voice" class="nav">받아 적기·말하기</button>
|
||||
<button data-screen="llm" class="nav">대화 모델</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="group" data-group="features">
|
||||
<button class="group-head" aria-expanded="false">기능</button>
|
||||
<div class="group-items">
|
||||
<button data-screen="features" class="nav">기능 켜기·MCP</button>
|
||||
<button data-screen="schedule" class="nav">예약</button>
|
||||
<button data-screen="collect" class="nav">백그라운드 수집</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="group" data-group="pc">
|
||||
<button class="group-head" aria-expanded="false">PC</button>
|
||||
<div class="group-items">
|
||||
<button data-screen="pcControl" class="nav">PC 제어·화면 보기</button>
|
||||
<button data-screen="game" class="nav">게임 모드</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="group" data-group="admin">
|
||||
<button class="group-head" aria-expanded="false">관리</button>
|
||||
<div class="group-items">
|
||||
<button data-screen="logs" class="nav">로그</button>
|
||||
<button data-screen="app" class="nav" id="nav-app">앱</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="side-foot" id="version"></div>
|
||||
</nav>
|
||||
<main id="screen" class="screen"></main>
|
||||
|
||||
@@ -17,6 +17,14 @@ button, input, select, textarea { font: inherit; color: inherit; }
|
||||
.nav { text-align: left; background: none; border: 0; padding: 8px 10px; border-radius: 6px; color: var(--ink2); cursor: pointer; }
|
||||
.nav:hover { background: var(--s2); }
|
||||
.nav.on { background: var(--s3); color: var(--ink); }
|
||||
.group { display: flex; flex-direction: column; margin-top: 6px; }
|
||||
.group-head { text-align: left; background: none; border: 0; padding: 8px 10px; border-radius: 6px; color: var(--ink3); cursor: pointer; font-size: 12px; font-weight: 600; letter-spacing: 0.2px; display: flex; align-items: center; gap: 6px; }
|
||||
.group-head::before { content: "▸"; font-size: 10px; color: var(--ink4); }
|
||||
.group-head[aria-expanded="true"]::before { content: "▾"; }
|
||||
.group-head:hover { background: var(--s2); color: var(--ink2); }
|
||||
.group-items { display: none; flex-direction: column; gap: 2px; }
|
||||
.group.open .group-items { display: flex; }
|
||||
.group-items .nav { padding-left: 24px; }
|
||||
.side-foot { margin-top: auto; color: var(--ink4); font-size: 12px; padding: 8px 10px; }
|
||||
.screen { padding: 28px 32px; max-width: 1100px; }
|
||||
h1 { font-size: 24px; font-weight: 600; letter-spacing: -0.4px; margin: 0 0 18px; }
|
||||
|
||||
@@ -26,7 +26,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
- **v2 schedule tool and scheduled tasks**: when the voice worker's brain sets a schedule backend (`turn_state.set_schedule`, every voice and chat reply), `schedule` is in the catalogue for the router (#7) and `prepare_intent()` (not force-offered every turn); without a backend it is not in the catalogue at all ([spec](src/jarvis/tools/builtin/schedule.spec.md)). A scheduled `task` the app fires is one more run of this loop (`reply` `via: "task"`, at the task's time, in the worker's chat lane, waiting for LLM switches like any reply): the asker's history scope, a fresh short-term memory, no prepared pick, no chat or schedule backend, `mcps={}` and `disabled_tools` widened to everything but `TASK_TOOLS` (`getWeather`, `webSearch`, `fetchWebPage`, `fetchMeals`, `collectedNews`, `toolSearchTool`), so the router, planner and loop see only those. The app waits at most 300 s for its answer and delivers it (`app/src/admin.spec.md`, "Schedules"). Eval: `evals/test_schedule_tool.py`.
|
||||
- **v2 personal notes**: the app sends the asker's standing instructions with every voice, chat and task reply; `run_reply_engine(person_notes=...)` adds them to the system message (after the warm profile, before the history block) inside a `<person_notes>` fence, labelled as that person's own answer preferences that never permit an action or a tool. Only the asker's notes, at most 20 of 300 characters. The `personalNotes` tool (add, list, delete) is in the catalogue only with a notes backend (`turn_state.set_notes`, every voice and chat reply; not tasks), also for `prepare_intent()`. No extra LLM call ([spec](src/jarvis/tools/builtin/personal_notes.spec.md)). Eval: `evals/test_personal_notes.py`.
|
||||
- **v2 collected news**: the `collectedNews` tool (list targets, read a target's newest items) puts the background collection's results into the loop inside an UNTRUSTED COLLECTED CONTENT fence; offered only with the app's collection file (`JARVIS_COLLECT_PATH`) and `features.collect` on, and one of the scheduled tasks' tools (`TASK_TOOLS`). No LLM call inside. The app's news notifications post titles and links with no LLM ([spec](src/jarvis/tools/builtin/collected_news.spec.md)). Eval: `evals/test_collected_news.py`.
|
||||
- **v2 screen view (pictures)**: the `screenView` tool (whole screen, one monitor or one window, or the list of monitors and windows) returns base64 JPEG pictures with its result; the loop puts them on the tool message (`images`) and `llm.py` sends them to Strata as a user message of `image_url` parts right after the tool messages (about 768 / 1280 px long side, or four parts for `detailed`; at most 3 MiB). Offered only when the app makes screen viewing available (Windows, someone registered for PC control), which also starts the LLM with `--vision gpu` (normal stage) or `cpu` (game stage), else `--vision off`. Pictures stay in the one reply: not in the tool carryover, logs or prompt dumps. No extra LLM call ([spec](src/jarvis/tools/builtin/screen_view.spec.md)). Eval: `evals/test_screen_view_tool.py`.
|
||||
- **v2 screen view (pictures)**: the `screenView` tool (whole screen, one monitor, one window or someone's Discord screen share in its popped-out window, or the list of monitors and windows) returns base64 JPEG pictures with its result; the loop puts them on the tool message (`images`) and `llm.py` sends them to Strata as a user message of `image_url` parts right after the tool messages (about 768 / 1280 px long side, or four parts for `detailed`; at most 3 MiB). Offered only when the app makes screen viewing available (Windows, someone registered for PC control), which also starts the LLM with `--vision gpu` (normal stage) or `cpu` (game stage), else `--vision off`. Pictures stay in the one reply: not in the tool carryover, logs or prompt dumps. No extra LLM call ([spec](src/jarvis/tools/builtin/screen_view.spec.md)). Eval: `evals/test_screen_view_tool.py`.
|
||||
- Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68))
|
||||
- Tool results from prior turns (raw or digested — see #5)
|
||||
- **Output**: OpenAI-style `{content, tool_calls, thinking}`. Consumed by the tool orchestrator and TTS pipeline. Natural-language content is delivered immediately; no post-turn evaluator runs. Spoken-answer length: the persona (`system_prompt.py`) and `voice_style` (`prompts/system.py`) both constrain the reply to a SINGLE sentence — any dry aside must fold into that one sentence as a trailing clause, never a second sentence. This keeps TTS latency down (synth time scales with text length) and matches the `agents/llm.md` operator instruction.
|
||||
|
||||
@@ -51,7 +51,7 @@ v1도 STT는 이미 faster-whisper입니다. 느린 진짜 이유는 엔진이
|
||||
[게임PC (Windows 11) 한 대에서 전부 실행]
|
||||
|
||||
1. Jarvis 관리 앱 (exe, Rust. 화면은 Tauri 2 + Windows 기본 WebView2)
|
||||
├─ GUI: 대시보드 / 봇 / 채널·호출 / 채팅 채널 / 대화 기억 / 모델 / 기능
|
||||
├─ GUI: 대시보드 / 대화 / 디스코드 / 대화 방식 / 음성·모델 / 기능 / PC / 관리
|
||||
│ / 게임 모드 / PC 제어 / 화면 보기 / 데이터 / 로그
|
||||
├─ 서비스 관리자: 아래 프로세스를 띄우고, 감시하고, 죽으면 재시작
|
||||
├─ 디스코드 음성 코어 (serenity + songbird + DAVE 복호화, Opus 해독은 앱이 직접)
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
|
||||
| 설정 | 내용 |
|
||||
|---|---|
|
||||
| 호출어(prefix) | GUI에서 여러 개 등록·수정·삭제 (예: "자비스", "야 자비스", "javis"). 음성과 채팅에 같이 쓰고, 채팅용은 따로 지정할 수도 있습니다. 비슷한 발음 허용 정도(예: "자비즈")도 조절합니다. |
|
||||
| 호출어(prefix) | GUI에서 여러 개 등록·수정·삭제. 기본값은 비서를 부르듯 "비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"입니다(사용자 요청, 2026-11-19, 그 전에는 "자비스"). 한 자리에서 여러 호출어가 들리면 가장 가깝고 긴 것을 부른 말로 보므로 "비서야, 멈춰"의 "야"가 남지 않습니다. 음성과 채팅에 같이 쓰고, 채팅용은 따로 지정할 수도 있습니다. 비슷한 발음 허용 정도(예: "자비즈")도 조절합니다. "비서"는 흔한 낱말이라 "우리 회사 비서가"처럼 대화 중에 나와도 부른 것으로 봅니다. |
|
||||
| 호출 방식 | (1) 호출어로 부를 때만 응답 **(기본값)** / (2) 봇에게 하는 말로 판단되면 응답 / (3) 모든 말에 응답 |
|
||||
| 호출어 뒤 쉼 | "자비스, … 지금 몇 시야?"처럼 이름 뒤에 쉬어서 "자비스"만 따로 들리면, 같은 사람이 이어서 하는 말(기본 1.5초 안에 시작)과 합쳐 한 질문으로 받습니다. 아무 말도 이어지지 않으면 "자비스"에만 대답합니다. 채팅은 합치지 않습니다. |
|
||||
| 지정한 사람 | GUI에서 서버 멤버 목록으로 고릅니다. 아래 두 설정을 **각각 켜고 끕니다**. |
|
||||
|
||||
@@ -47,6 +47,8 @@
|
||||
|
||||
## 설정 화면
|
||||
|
||||
왼쪽 메뉴는 대시보드와 대화를 위에 두고, 나머지는 묶음으로 접어 둡니다(사용자 요청, 2026-11-19). 묶음 이름을 누르면 펼쳐집니다: 디스코드(봇, 음성 채널, 채팅 채널), 대화 방식(호출, 여럿이 말할 때, 대화 기억, 사람별 지침), 음성·모델(받아 적기·말하기, 대화 모델), 기능(기능 켜기·MCP, 예약, 백그라운드 수집), PC(PC 제어·화면 보기, 게임 모드), 관리(로그, 앱). 아래 표의 "채널·호출"은 음성 채널, 호출, 여럿이 말할 때 세 화면으로, "모델"은 받아 적기·말하기와 대화 모델로 나눴습니다. 화면마다 정확히 어떤 항목이 있는지는 [admin.spec.md](../../app/src/admin.spec.md) "Screens"에 있습니다.
|
||||
|
||||
| 화면 | 내용 |
|
||||
|---|---|
|
||||
| 대시보드 | 서비스 상태(디스코드, LLM, STT, TTS), GPU/VRAM/RAM 사용량, 현재 음성 채널과 참가자, 실시간 자막, 단계별 지연 시간, 성능 단계(평소/게임 중/일시정지와 이유), 놓친 요청, 봇 시작/정지 |
|
||||
|
||||
@@ -319,6 +319,8 @@ LLM과 STT를 함께 켠 측정(M0 `gamemode_stt`, 게임 없음). STT를 먼저
|
||||
|
||||
최소화하면 캡처가 안 되므로 그때는 창을 다시 열어 달라고 안내합니다.
|
||||
|
||||
**지금 구현(M7 3단계 둘째 부분, 2026-11-19)**: `screenView`의 `share` 대상(사람 이름은 선택), 앱 `app/src/screen.rs`와 `discord.rs`([admin.spec.md](../../app/src/admin.spec.md) "Screen viewing"). 분리 창은 제목이 "Discord"로 끝나지 않는 디스코드 창으로 찾고, 자비스가 있는 음성 채널 이름이 제목에 든 창을 먼저 씁니다. 창이 없으면 음성 채널에 "디스코드에서 철수님 화면공유를 시청하고 창 분리 버튼을 눌러 주세요."라고 말하고 1초마다 30초 동안 찾습니다(채팅으로 물으면 기다리지 않고 같은 안내로 답함). 대기 시간은 아직 설정 화면에 두지 않았습니다. 실제 디스코드 분리 창 제목과 실제 모델(Qwen3.6)의 도구 고르기는 게임PC에서 확인할 것. 작은 모델(gemma3:4b, qwen3:8b)로 돌린 평가에서는 "철수 화면 뭐 하고 있어?"에 `share`를 고르는 일이 드물었습니다.
|
||||
|
||||
## 백그라운드 크로미움으로 정보 수집
|
||||
|
||||
- 봇용 크롬(데이터 폴더 안)을 크롬 개발자 프로토콜(CDP)로 조종합니다. 웹 검색, 페이지 읽기, 예약 수집에 씁니다.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -6,6 +6,8 @@ Eval: seeing the screen (src/jarvis/tools/builtin/screen_view.spec.md).
|
||||
- When the app refuses (the person is not registered), the answer says it cannot see the screen and
|
||||
does not describe one.
|
||||
- A question that is not about the screen does not capture it.
|
||||
- "철수 화면 뭐 하고 있어?" asks for 철수's Discord screen share (target share, person 철수), and
|
||||
the answer reads that picture.
|
||||
|
||||
The screen backend is a fake that returns a fixed picture; the tool, the reply loop and the picture
|
||||
transport (llm.py) run for real. Needs a model that reads pictures:
|
||||
@@ -38,8 +40,8 @@ class FakeScreen:
|
||||
{"number": 2, "width": 1280, "height": 720, "main": False}],
|
||||
"windows": [{"program": "notepad", "title": "Untitled - Notepad",
|
||||
"minimised": False}]}}
|
||||
return {"ok": True, "item": {"what": f"모니터 {kw.get('monitor', 2)}", "width": 1280, "height": 720,
|
||||
"parts": 1}, "images": [PICTURE]}
|
||||
what = "화면공유" if kw.get("target") == "share" else f"모니터 {kw.get('monitor', 2)}"
|
||||
return {"ok": True, "item": {"what": what, "width": 1280, "height": 720, "parts": 1}, "images": [PICTURE]}
|
||||
|
||||
|
||||
def _run(mock_config, eval_db, eval_dialogue_memory, query, screen):
|
||||
@@ -84,3 +86,11 @@ class TestScreenView:
|
||||
screen = FakeScreen()
|
||||
_run(mock_config, eval_db, eval_dialogue_memory, "고양이는 하루에 몇 시간 자?", screen)
|
||||
assert not any(a == "capture" for a, _ in screen.acts), screen.acts
|
||||
|
||||
def test_someones_screen_share_is_asked_for_by_their_name(self, mock_config, eval_db, eval_dialogue_memory):
|
||||
screen = FakeScreen()
|
||||
reply = _run(mock_config, eval_db, eval_dialogue_memory, "철수 화면 뭐 하고 있어?", screen)
|
||||
captures = [kw for a, kw in screen.acts if a == "capture"]
|
||||
assert captures and captures[0].get("target") == "share" and "철수" in (captures[0].get("person") or ""), \
|
||||
screen.acts
|
||||
assert "BANANA" in reply.upper() or "바나나" in reply or "42" in reply, reply
|
||||
|
||||
@@ -10,9 +10,10 @@ from ..base import Tool, ToolContext
|
||||
from ..types import ToolExecutionResult
|
||||
from ...debug import debug_log
|
||||
|
||||
_TARGETS = ("screen", "monitor", "window", "list")
|
||||
_TARGETS = ("screen", "monitor", "window", "share", "list")
|
||||
_DETAILS = ("fast", "normal", "detailed")
|
||||
MAX_WINDOW_CHARS = 200
|
||||
MAX_PERSON_CHARS = 200
|
||||
|
||||
|
||||
def _fail(text: str) -> ToolExecutionResult:
|
||||
@@ -37,10 +38,11 @@ class ScreenViewTool(Tool):
|
||||
def description(self) -> str:
|
||||
return (
|
||||
"See this PC's screen as a picture: target='screen' (all monitors), 'monitor' with its "
|
||||
"number, or 'window' with part of the program's name or window title; target='list' "
|
||||
"names the monitors and open windows. Use it whenever the person asks what is on their "
|
||||
"screen, a monitor or a program's window; detail='detailed' to read small text. Answer "
|
||||
"only from the picture."
|
||||
"number, 'window' with part of the program's name or window title, or 'share' for "
|
||||
"another person's screen, which they share on Discord (person: their name); "
|
||||
"target='list' names the monitors and open windows. Use it whenever the person asks "
|
||||
"what is on their screen, a monitor, a program's window or someone else's screen; "
|
||||
"detail='detailed' to read small text. Answer only from the picture."
|
||||
)
|
||||
|
||||
@property
|
||||
@@ -51,6 +53,7 @@ class ScreenViewTool(Tool):
|
||||
"target": {"type": "string", "enum": list(_TARGETS)},
|
||||
"monitor": {"type": "integer", "description": "monitor: its number (1, 2, ...)"},
|
||||
"window": {"type": "string", "description": "window: part of its program name or title"},
|
||||
"person": {"type": "string", "description": "share: the name of the person sharing"},
|
||||
"detail": {"type": "string", "enum": list(_DETAILS)},
|
||||
},
|
||||
"required": [],
|
||||
@@ -80,6 +83,12 @@ class ScreenViewTool(Tool):
|
||||
if not w or len(w) > MAX_WINDOW_CHARS:
|
||||
return _fail("window needs part of the program name or title; call target='list' to see them")
|
||||
fields["window"] = w
|
||||
if target == "share":
|
||||
p = " ".join(str(args.get("person") or "").split())
|
||||
if len(p) > MAX_PERSON_CHARS:
|
||||
return _fail("person needs the name of the person sharing their screen")
|
||||
if p:
|
||||
fields["person"] = p
|
||||
fields["detail"] = _word(args.get("detail"), _DETAILS, "normal")
|
||||
result = backend.act("capture", **fields) or {}
|
||||
images = [i for i in result.get("images") or [] if isinstance(i, str) and i]
|
||||
|
||||
@@ -32,6 +32,7 @@ Lets Jarvis see this PC's screen: all monitors as one picture, one monitor, or o
|
||||
| `screen` (default) | | every monitor as one picture |
|
||||
| `monitor` | `monitor` (its number, as Windows numbers it) | that monitor |
|
||||
| `window` | `window` (part of its title or program name) | that window, also when other windows cover it |
|
||||
| `share` | `person` (optional: the name of the person sharing, as said) | the Discord screen share someone streams in Jarvis's voice channel, from the window Discord pops it out to |
|
||||
| `list` | | the monitors (number, size, main) and the open windows (program, title), no picture |
|
||||
|
||||
`detail`: `fast` (long side about 768 px), `normal` (default, about 1280 px) or `detailed` (the
|
||||
@@ -44,6 +45,9 @@ as their first known word; unknown values fall back to the defaults.
|
||||
|
||||
- `act("capture", target=..., monitor=..., window=..., detail=...)` gives `{"ok": true, "item":
|
||||
{"what", "width", "height", "parts"}, "images": ["<base64 JPEG>", ...]}`.
|
||||
- `act("capture", target="share", person=..., detail=...)` gives the same; `person` is sent only
|
||||
when given (at most 200 characters). The app checks that the person shares their screen and may
|
||||
wait up to 30 s while it asks for the share to be popped out (admin.spec.md, "Screen viewing").
|
||||
- `act("list")` gives `{"ok": true, "item": {"monitors": [...], "windows": [...]}}`.
|
||||
- Failures are `{"ok": false, "error": "..."}`.
|
||||
|
||||
|
||||
@@ -169,8 +169,9 @@ class VoiceWorker:
|
||||
|
||||
def action_timeout_s(self, kind: str) -> float:
|
||||
"""How long an app action is waited for: a PC action (starting a program and reading its new
|
||||
window, admin.spec.md "PC control") gets at least PC_ACTION_TIMEOUT_S."""
|
||||
return max(self.chat_timeout_s, PC_ACTION_TIMEOUT_S) if kind == "pc" else self.chat_timeout_s
|
||||
window, admin.spec.md "PC control") and a screen action (a screen share waited for while it
|
||||
is popped out, "Screen viewing") get at least PC_ACTION_TIMEOUT_S."""
|
||||
return max(self.chat_timeout_s, PC_ACTION_TIMEOUT_S) if kind in ("pc", "screen") else self.chat_timeout_s
|
||||
|
||||
async def start(self) -> None:
|
||||
if self.history is not None:
|
||||
|
||||
@@ -436,7 +436,8 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
|
||||
|
||||
```
|
||||
<- {"type": "screen_action", "id": "s1", "action": "capture|list", "user": "<id>", "via": "voice|chat",
|
||||
"channel": "...", "target": "screen|monitor|window", "monitor": 2, "window": "chrome", "detail": "fast|normal|detailed"}
|
||||
"channel": "...", "target": "screen|monitor|window|share", "monitor": 2, "window": "chrome", "person": "철수",
|
||||
"detail": "fast|normal|detailed"}
|
||||
-> {"type": "screen_action_result", "id": "s1", "ok": true, "item": {...}, "images": ["<base64 JPEG>"]} // or ok false with "error"
|
||||
```
|
||||
|
||||
@@ -444,6 +445,8 @@ captures and decides who may see (`app/src/admin.spec.md`, "Screen viewing"):
|
||||
waiting are those of chat and schedule actions. `images` that are not a list of strings are
|
||||
dropped. The pictures stay in that one reply (screen_view.spec.md).
|
||||
- Messages may be up to 4 MiB (pictures), the WebSocket's limit.
|
||||
- A screen action is waited for as long as a PC action (45 s): the app may wait up to 30 s for a
|
||||
screen share to be popped out.
|
||||
|
||||
### PC control (M7)
|
||||
|
||||
|
||||
@@ -68,6 +68,15 @@ def test_a_window_is_found_by_part_of_its_name(app):
|
||||
assert app.calls[-1] == ("capture", {"target": "window", "window": "크롬", "detail": "normal"})
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_discord_screen_share_is_asked_for_with_the_person_named(app):
|
||||
assert run(target="share", person=" 철수 ").success
|
||||
assert run(target="share").success
|
||||
assert app.calls == [("capture", {"target": "share", "person": "철수", "detail": "normal"}),
|
||||
("capture", {"target": "share", "detail": "normal"})]
|
||||
assert not run(target="share", person="x" * 201).success and len(app.calls) == 2
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("args", [{"target": "monitor"}, {"target": "monitor", "monitor": 0},
|
||||
{"target": "monitor", "monitor": "two"}, {"target": "window"},
|
||||
|
||||
@@ -1652,26 +1652,32 @@ def test_a_pc_action_goes_to_the_app_and_an_ask_answer_keeps_what_to_ask(store):
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_pc_action_waits_longer_than_the_other_app_actions(store):
|
||||
def test_pc_and_screen_actions_wait_longer_than_the_other_app_actions(store):
|
||||
"""Starting a program and reading its new window takes the app up to about 20 s (admin.spec.md,
|
||||
"PC control"): a PC action is not given up after the chat actions' wait."""
|
||||
"PC control"), and a screen share is waited for up to 30 s while it is popped out ("Screen
|
||||
viewing"): those actions are not given up after the chat actions' wait."""
|
||||
results = []
|
||||
brain = FakeBrain(lambda u, t: results.append((brain.screen.act("list"), brain.pc.act("run", program="calc")))
|
||||
or "ok")
|
||||
brain = FakeBrain(lambda u, t: results.append((brain.notes_backend.act("list"),
|
||||
brain.screen.act("capture", target="share"),
|
||||
brain.pc.act("run", program="calc"))) or "ok")
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws)
|
||||
await send(ws, type="reply", id="r1", user="100", text="q", via="voice", speak=False)
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "screen_action")
|
||||
seen = await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "pc_action")
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "note_action")
|
||||
seen = await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "screen_action")
|
||||
await asyncio.sleep(0.6) # longer than chat_timeout_s
|
||||
await send(ws, type="screen_action_result", id=seen[-1]["id"], ok=True, item={"what": "화면공유"},
|
||||
images=["x"])
|
||||
seen = await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "pc_action")
|
||||
await asyncio.sleep(0.6)
|
||||
await send(ws, type="pc_action_result", id=seen[-1]["id"], ok=True, item={"window": {"id": "w1"}})
|
||||
await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
|
||||
asyncio.run(with_worker(body, store=store, brain=brain, chat_timeout_s=0.3))
|
||||
(screen, pc), = results
|
||||
assert not screen["ok"] and pc == {"ok": True, "item": {"window": {"id": "w1"}}}
|
||||
(note, screen, pc), = results
|
||||
assert not note["ok"] and screen["ok"] and pc == {"ok": True, "item": {"window": {"id": "w1"}}}
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
|
||||
Reference in New Issue
Block a user