Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
140fc56f18 | ||
|
|
5ee47827f3 | ||
|
|
c189ce2e65 | ||
|
|
086dd5cde7 |
@@ -1,6 +1,6 @@
|
|||||||
Data privacy comes first, always.
|
Data privacy comes first, always.
|
||||||
|
|
||||||
All user-facing command line output should make use of emojis. Especially an initial emoji to start off the lines that depict what the line is about. Output should make use of indentation spacing to establish a visual hierarchy and aim to make output as easy to sift through as possible. Exception: Windows .bat scripts cannot use emojis (cmd.exe doesn't render Unicode properly).
|
This assistant is used through a Discord bot with voice (TTS) replies, not a CLI. Do not add emojis to user-facing assistant output. Keep output plain and readable. (Runtime assistant behaviour lives in `agents/*.md`, which is injected into the reply LLM's prompt.)
|
||||||
|
|
||||||
Any important point in our logical flows should have debug logs using the `debug_log` method from `src/jarvis/debug.py`. Avoid excessive logging to keep the logs easily readable and actionable.
|
Any important point in our logical flows should have debug logs using the `debug_log` method from `src/jarvis/debug.py`. Avoid excessive logging to keep the logs easily readable and actionable.
|
||||||
|
|
||||||
|
|||||||
13
agents/llm.md
Normal file
13
agents/llm.md
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
# 자비스 운영자 지시
|
||||||
|
|
||||||
|
- 너의 이름은 자비스다.
|
||||||
|
- 모든 답변은 음성(TTS)으로 읽혀 나간다. 그러니 최대한 간결하게, 한두 문장으로 답한다. 목록, 마크다운, 이모지, 그리고 소리 내어 읽기 어려운 특수문자는 쓰지 않는다.
|
||||||
|
- 정해진 문구에만 반응하지 말고, 실제 사람처럼 말의 뉘앙스와 맥락으로 의도를 알아듣고 처리한다.
|
||||||
|
|
||||||
|
화면 속 크롬(방송 화면)에서 유튜브를 다룰 때 (화면에 보여야 하므로 반드시 on-screen 브라우저 제어 도구로 수행한다):
|
||||||
|
|
||||||
|
- "유튜브 켜줘" → 방송 크롬에서 유튜브를 연다.
|
||||||
|
- "유튜브에서 OO 검색해줘" → 유튜브로 가서 검색창에 OO를 사람이 직접 타이핑하듯 입력하고 검색한다.
|
||||||
|
- "위에서 N번째 영상 재생해줘" 또는 "왼쪽에서 N번째 영상 재생해줘" → 검색 결과 목록에서 그 위치의 영상을 재생한다.
|
||||||
|
- "일시정지해줘" → 현재 영상을 일시정지한다. "다시 재생해줘" → 이어서 재생한다.
|
||||||
|
- "영상 종료" 또는 "그만 보여줘" → 뒤로 가서 직전 화면으로 돌아간다.
|
||||||
@@ -2,10 +2,11 @@
|
|||||||
// 9222) so the action is visible on the Go-Live broadcast, and prints a JSON
|
// 9222) so the action is visible on the Go-Live broadcast, and prints a JSON
|
||||||
// result on stdout for the Python `browseAndSearch` tool to wrap.
|
// result on stdout for the Python `browseAndSearch` tool to wrap.
|
||||||
//
|
//
|
||||||
// node browse-search.mjs "<query>" [search|youtube]
|
// node browse-search.mjs "<query>" [search|youtube] [index]
|
||||||
//
|
//
|
||||||
// - search : Google-search the query, return the top organic results.
|
// - search : Google-search the query, return the top organic results.
|
||||||
// - youtube : search YouTube and play the first result.
|
// - youtube : search YouTube and play a result. `index` is the 1-based position
|
||||||
|
// from the top of the result list (default 1 = first result).
|
||||||
//
|
//
|
||||||
// Backend selection for `search`:
|
// Backend selection for `search`:
|
||||||
// 1. The broadcast Chrome over CDP (visible on the Go-Live stream).
|
// 1. The broadcast Chrome over CDP (visible on the Go-Live stream).
|
||||||
@@ -29,6 +30,9 @@ const UA =
|
|||||||
'(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36';
|
'(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36';
|
||||||
const query = process.argv[2] || '';
|
const query = process.argv[2] || '';
|
||||||
const mode = (process.argv[3] || 'search').toLowerCase();
|
const mode = (process.argv[3] || 'search').toLowerCase();
|
||||||
|
// 1-based position of the YouTube result to play, counted from the top of the
|
||||||
|
// list. Defaults to 1 (first result). Anything <1 or non-numeric falls back to 1.
|
||||||
|
const playIndex = Math.max(1, parseInt(process.argv[4], 10) || 1);
|
||||||
const out = (o) => { process.stdout.write(JSON.stringify(o)); };
|
const out = (o) => { process.stdout.write(JSON.stringify(o)); };
|
||||||
|
|
||||||
if (!query) { out({ ok: false, error: 'no query' }); process.exit(1); }
|
if (!query) { out({ ok: false, error: 'no query' }); process.exit(1); }
|
||||||
@@ -105,15 +109,21 @@ try {
|
|||||||
await page.bringToFront().catch(() => {});
|
await page.bringToFront().catch(() => {});
|
||||||
|
|
||||||
if (mode === 'youtube') {
|
if (mode === 'youtube') {
|
||||||
// Type into YouTube's search box like a person, then play the first result.
|
// Type into YouTube's search box like a person, then play the requested
|
||||||
|
// result (the Nth from the top of the list; default the first).
|
||||||
await typeSearch('https://www.youtube.com/?hl=ko', 'input#search, input[name="search_query"]', query);
|
await typeSearch('https://www.youtube.com/?hl=ko', 'input#search, input[name="search_query"]', query);
|
||||||
await page.waitForSelector('ytd-video-renderer a#video-title, a#video-title', { timeout: 20000 });
|
await page.waitForSelector('ytd-video-renderer a#video-title, a#video-title', { timeout: 20000 });
|
||||||
const first = page.locator('ytd-video-renderer a#video-title, a#video-title').first();
|
const results = page.locator('ytd-video-renderer a#video-title, a#video-title');
|
||||||
const title = (await first.getAttribute('title').catch(() => '')) || (await first.innerText().catch(() => ''));
|
// Clamp to what's actually on the page so "play the 5th" still plays the
|
||||||
await first.click();
|
// last available result rather than failing when fewer were returned.
|
||||||
|
const available = await results.count();
|
||||||
|
const targetIdx = Math.min(playIndex, Math.max(available, 1)) - 1;
|
||||||
|
const target = results.nth(targetIdx);
|
||||||
|
const title = (await target.getAttribute('title').catch(() => '')) || (await target.innerText().catch(() => ''));
|
||||||
|
await target.click();
|
||||||
await page.waitForSelector('#movie_player', { timeout: 20000 });
|
await page.waitForSelector('#movie_player', { timeout: 20000 });
|
||||||
await page.evaluate(() => { const v = document.querySelector('video'); if (v && v.paused) v.play(); });
|
await page.evaluate(() => { const v = document.querySelector('video'); if (v && v.paused) v.play(); });
|
||||||
out({ ok: true, mode, title: (title || '').trim(), url: page.url() });
|
out({ ok: true, mode, index: targetIdx + 1, title: (title || '').trim(), url: page.url() });
|
||||||
} else {
|
} else {
|
||||||
// Type into Google's search box like a person, then read the results.
|
// Type into Google's search box like a person, then read the results.
|
||||||
await typeSearch('https://www.google.com/?hl=ko', 'textarea[name="q"], input[name="q"]', query);
|
await typeSearch('https://www.google.com/?hl=ko', 'textarea[name="q"], input[name="q"]', query);
|
||||||
|
|||||||
@@ -16,6 +16,9 @@ set -euo pipefail
|
|||||||
# by default so everything runs on one resident model; override if you pull a
|
# by default so everything runs on one resident model; override if you pull a
|
||||||
# dedicated small model.
|
# dedicated small model.
|
||||||
: "${OLLAMA_INTENT_MODEL:=${OLLAMA_CHAT_MODEL}}"
|
: "${OLLAMA_INTENT_MODEL:=${OLLAMA_CHAT_MODEL}}"
|
||||||
|
# Cap chat-model output tokens per turn (worst-case latency guard). Spoken
|
||||||
|
# answers are 1-2 sentences; 512 is safe headroom above tool-call JSON. 0 = off.
|
||||||
|
: "${OLLAMA_NUM_PREDICT:=512}"
|
||||||
: "${OLLAMA_EMBED_MODEL:=nomic-embed-text}"
|
: "${OLLAMA_EMBED_MODEL:=nomic-embed-text}"
|
||||||
: "${WHISPER_MODEL:=small}"
|
: "${WHISPER_MODEL:=small}"
|
||||||
: "${WHISPER_DEVICE:=cuda}"
|
: "${WHISPER_DEVICE:=cuda}"
|
||||||
@@ -32,7 +35,7 @@ set -euo pipefail
|
|||||||
: "${XDG_RUNTIME_DIR:=/run/user/0}"
|
: "${XDG_RUNTIME_DIR:=/run/user/0}"
|
||||||
: "${PULSE_SERVER:=unix:${XDG_RUNTIME_DIR}/pulse/native}"
|
: "${PULSE_SERVER:=unix:${XDG_RUNTIME_DIR}/pulse/native}"
|
||||||
|
|
||||||
export VNC_RESOLUTION OLLAMA_BASE_URL OLLAMA_CHAT_MODEL OLLAMA_INTENT_MODEL OLLAMA_EMBED_MODEL \
|
export VNC_RESOLUTION OLLAMA_BASE_URL OLLAMA_CHAT_MODEL OLLAMA_NUM_PREDICT OLLAMA_INTENT_MODEL OLLAMA_EMBED_MODEL \
|
||||||
WHISPER_MODEL WHISPER_DEVICE WHISPER_COMPUTE_TYPE JARVIS_DB_PATH \
|
WHISPER_MODEL WHISPER_DEVICE WHISPER_COMPUTE_TYPE JARVIS_DB_PATH \
|
||||||
PIPER_VOICE PIPER_VOICE_DIR TTS_PIPER_MODEL_PATH BRIDGE_HOST BRIDGE_PORT \
|
PIPER_VOICE PIPER_VOICE_DIR TTS_PIPER_MODEL_PATH BRIDGE_HOST BRIDGE_PORT \
|
||||||
XDG_RUNTIME_DIR PULSE_SERVER
|
XDG_RUNTIME_DIR PULSE_SERVER
|
||||||
@@ -51,12 +54,18 @@ export JARVIS_CONFIG_PATH=/app/config/jarvis.json
|
|||||||
# the env-rendered config, so changes survive container recreate.
|
# the env-rendered config, so changes survive container recreate.
|
||||||
if [ -f /data/jarvis-settings.json ]; then
|
if [ -f /data/jarvis-settings.json ]; then
|
||||||
python3 - <<'PY' || true
|
python3 - <<'PY' || true
|
||||||
import json
|
import json, os
|
||||||
try:
|
try:
|
||||||
base = json.load(open("/app/config/jarvis.json"))
|
base = json.load(open("/app/config/jarvis.json"))
|
||||||
ov = json.load(open("/data/jarvis-settings.json"))
|
ov = json.load(open("/data/jarvis-settings.json"))
|
||||||
if isinstance(base, dict) and isinstance(ov, dict):
|
if isinstance(base, dict) and isinstance(ov, dict):
|
||||||
base.update(ov)
|
base.update(ov)
|
||||||
|
# A stale persisted tts_engine from an earlier voice (melo/xtts, no
|
||||||
|
# longer built into the image) would override the configured engine and
|
||||||
|
# leave the bot silent. Reset those to the env-configured engine.
|
||||||
|
if base.get("tts_engine") in ("melo", "xtts"):
|
||||||
|
base["tts_engine"] = os.environ.get("TTS_ENGINE", "edge")
|
||||||
|
print(f"[entrypoint] reset stale tts_engine -> {base['tts_engine']}")
|
||||||
json.dump(base, open("/app/config/jarvis.json", "w"), ensure_ascii=False, indent=2)
|
json.dump(base, open("/app/config/jarvis.json", "w"), ensure_ascii=False, indent=2)
|
||||||
print("[entrypoint] merged persistent settings overrides")
|
print("[entrypoint] merged persistent settings overrides")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
|||||||
@@ -4,6 +4,7 @@
|
|||||||
"ollama_base_url": "${OLLAMA_BASE_URL}",
|
"ollama_base_url": "${OLLAMA_BASE_URL}",
|
||||||
"ollama_embed_model": "${OLLAMA_EMBED_MODEL}",
|
"ollama_embed_model": "${OLLAMA_EMBED_MODEL}",
|
||||||
"ollama_chat_model": "${OLLAMA_CHAT_MODEL}",
|
"ollama_chat_model": "${OLLAMA_CHAT_MODEL}",
|
||||||
|
"ollama_num_predict": "${OLLAMA_NUM_PREDICT}",
|
||||||
"intent_judge_model": "${OLLAMA_INTENT_MODEL}",
|
"intent_judge_model": "${OLLAMA_INTENT_MODEL}",
|
||||||
"tts_enabled": true,
|
"tts_enabled": true,
|
||||||
"tts_engine": "${TTS_ENGINE}",
|
"tts_engine": "${TTS_ENGINE}",
|
||||||
|
|||||||
@@ -18,6 +18,27 @@ cat > /etc/opt/chrome/policies/managed/jarvis.json <<'JSON'
|
|||||||
{ "CommandLineFlagSecurityWarningsEnabled": false }
|
{ "CommandLineFlagSecurityWarningsEnabled": false }
|
||||||
JSON
|
JSON
|
||||||
|
|
||||||
|
# Seed the profile's web-content language to Korean so sites (YouTube, Google,
|
||||||
|
# Naver) render in Korean. --lang sets Chrome's own UI, but the Accept-Language
|
||||||
|
# sent to sites comes from the profile's intl.accept_languages, which a persisted
|
||||||
|
# user-data-dir would otherwise keep at en-US regardless of --accept-lang.
|
||||||
|
PREFS_DIR="${CHROME_PROFILE_DIR:-/root/chrome-profile}/Default"
|
||||||
|
PREFS="${PREFS_DIR}/Preferences"
|
||||||
|
mkdir -p "$PREFS_DIR"
|
||||||
|
if [ -f "$PREFS" ]; then
|
||||||
|
python3 - "$PREFS" <<'PY' 2>/dev/null || true
|
||||||
|
import json, sys
|
||||||
|
p = sys.argv[1]
|
||||||
|
d = json.load(open(p))
|
||||||
|
d.setdefault("intl", {})
|
||||||
|
d["intl"]["accept_languages"] = "ko-KR,ko"
|
||||||
|
d["intl"]["selected_languages"] = "ko-KR,ko"
|
||||||
|
json.dump(d, open(p, "w"), ensure_ascii=False)
|
||||||
|
PY
|
||||||
|
else
|
||||||
|
printf '%s' '{"intl":{"accept_languages":"ko-KR,ko","selected_languages":"ko-KR,ko"}}' > "$PREFS"
|
||||||
|
fi
|
||||||
|
|
||||||
# Minimal, non-automation flags. --remote-debugging exposes CDP so the brain can
|
# Minimal, non-automation flags. --remote-debugging exposes CDP so the brain can
|
||||||
# drive this on-screen Chrome (Google/YouTube/Naver), --disable-features=Translate
|
# drive this on-screen Chrome (Google/YouTube/Naver), --disable-features=Translate
|
||||||
# hides the translate popup. NO --test-type / --disable-blink-features.
|
# hides the translate popup. NO --test-type / --disable-blink-features.
|
||||||
@@ -26,6 +47,7 @@ exec google-chrome \
|
|||||||
--no-default-browser-check \
|
--no-default-browser-check \
|
||||||
--disable-features=Translate,TranslateUI \
|
--disable-features=Translate,TranslateUI \
|
||||||
--lang=ko-KR \
|
--lang=ko-KR \
|
||||||
|
--accept-lang=ko-KR,ko \
|
||||||
--remote-debugging-port="${CDP_PORT:-9222}" \
|
--remote-debugging-port="${CDP_PORT:-9222}" \
|
||||||
--remote-debugging-address="${CDP_BIND:-127.0.0.1}" \
|
--remote-debugging-address="${CDP_BIND:-127.0.0.1}" \
|
||||||
--user-data-dir="${CHROME_PROFILE_DIR:-/root/chrome-profile}" \
|
--user-data-dir="${CHROME_PROFILE_DIR:-/root/chrome-profile}" \
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
|||||||
- Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68))
|
- Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68))
|
||||||
- Tool results from prior turns (raw or digested — see #5)
|
- Tool results from prior turns (raw or digested — see #5)
|
||||||
- **Output**: OpenAI-style `{content, tool_calls, thinking}`. Consumed by the tool orchestrator and TTS pipeline. Natural-language content is delivered immediately; no post-turn evaluator runs.
|
- **Output**: OpenAI-style `{content, tool_calls, thinking}`. Consumed by the tool orchestrator and TTS pipeline. Natural-language content is delivered immediately; no post-turn evaluator runs.
|
||||||
- **Limits**: `num_ctx: 8192` (explicit). Timeout `llm_chat_timeout_sec` (45s). Auto-fallback from native to text tool-calls on HTTP 400 (`ToolsNotSupportedError`), sticky for the session. Risk: `fetch_web_page` truncates at 50,000 chars (~37k tokens) — mitigated for SMALL models by tool-result digest (#5) which compresses the payload before it enters the messages history. LARGE models receive the raw payload and may silently see a truncated context.
|
- **Limits**: `num_ctx: 8192` (explicit). Output `num_predict: cfg.ollama_num_predict` (default 512, `0`/negative disables) caps generated tokens per turn — a worst-case latency guard for short spoken answers; the headroom stays above tool-call JSON so it does not truncate tool calls (both native and text tool-call paths). Timeout `llm_chat_timeout_sec` (45s). Auto-fallback from native to text tool-calls on HTTP 400 (`ToolsNotSupportedError`), sticky for the session. Risk: `fetch_web_page` truncates at 50,000 chars (~37k tokens) — mitigated for SMALL models by tool-result digest (#5) which compresses the payload before it enters the messages history. LARGE models receive the raw payload and may silently see a truncated context.
|
||||||
|
|
||||||
## 2. Intent Judge
|
## 2. Intent Judge
|
||||||
|
|
||||||
|
|||||||
@@ -85,6 +85,12 @@ class Settings:
|
|||||||
llm_digest_timeout_sec: float
|
llm_digest_timeout_sec: float
|
||||||
llm_embedding_timeout_sec: float
|
llm_embedding_timeout_sec: float
|
||||||
llm_profile_select_timeout_sec: float
|
llm_profile_select_timeout_sec: float
|
||||||
|
# Upper bound on tokens the chat model may generate per reply turn. Spoken
|
||||||
|
# (TTS) answers are 1-2 sentences, so a cap bounds the worst-case latency of
|
||||||
|
# a model that occasionally rambles or loops without changing normal answers.
|
||||||
|
# The headroom (default 512) sits well above this app's short tool-call JSON,
|
||||||
|
# so capping never truncates a tool call. 0 (or negative) disables the cap.
|
||||||
|
ollama_num_predict: int
|
||||||
|
|
||||||
# Profiles & Behavior
|
# Profiles & Behavior
|
||||||
active_profiles: list[str]
|
active_profiles: list[str]
|
||||||
@@ -394,6 +400,9 @@ def get_default_config() -> Dict[str, Any]:
|
|||||||
"llm_digest_timeout_sec": 8.0,
|
"llm_digest_timeout_sec": 8.0,
|
||||||
"llm_embedding_timeout_sec": 60.0,
|
"llm_embedding_timeout_sec": 60.0,
|
||||||
"llm_profile_select_timeout_sec": 30.0,
|
"llm_profile_select_timeout_sec": 30.0,
|
||||||
|
# Cap on chat-model output tokens per turn (worst-case latency guard).
|
||||||
|
# 512 is safe headroom above short TTS answers and tool-call JSON; 0 off.
|
||||||
|
"ollama_num_predict": 512,
|
||||||
|
|
||||||
# Profiles & Behavior
|
# Profiles & Behavior
|
||||||
"active_profiles": ["developer", "business", "life"],
|
"active_profiles": ["developer", "business", "life"],
|
||||||
@@ -608,7 +617,11 @@ def load_settings() -> Settings:
|
|||||||
active_profiles = _ensure_list(merged.get("active_profiles"))
|
active_profiles = _ensure_list(merged.get("active_profiles"))
|
||||||
tts_enabled = bool(merged.get("tts_enabled", True))
|
tts_enabled = bool(merged.get("tts_enabled", True))
|
||||||
tts_engine = str(merged.get("tts_engine", "piper")).lower()
|
tts_engine = str(merged.get("tts_engine", "piper")).lower()
|
||||||
if tts_engine not in ("piper", "chatterbox"):
|
# "edge" (Microsoft Edge TTS) is the containerized bridge's Korean voice;
|
||||||
|
# "melo" is the legacy warm-worker voice. Both are multilingual, so they must
|
||||||
|
# be preserved here — coercing them to "piper" would mislabel the engine as
|
||||||
|
# English-only in reply_language_directive().
|
||||||
|
if tts_engine not in ("piper", "chatterbox", "edge", "melo"):
|
||||||
tts_engine = "piper" # Default to piper if invalid value
|
tts_engine = "piper" # Default to piper if invalid value
|
||||||
tts_voice_val = merged.get("tts_voice")
|
tts_voice_val = merged.get("tts_voice")
|
||||||
tts_voice = None if tts_voice_val in (None, "", "null") else str(tts_voice_val)
|
tts_voice = None if tts_voice_val in (None, "", "null") else str(tts_voice_val)
|
||||||
@@ -759,6 +772,10 @@ def load_settings() -> Settings:
|
|||||||
llm_digest_timeout_sec = float(merged.get("llm_digest_timeout_sec", 8.0))
|
llm_digest_timeout_sec = float(merged.get("llm_digest_timeout_sec", 8.0))
|
||||||
llm_embedding_timeout_sec = float(merged.get("llm_embedding_timeout_sec", 60.0))
|
llm_embedding_timeout_sec = float(merged.get("llm_embedding_timeout_sec", 60.0))
|
||||||
llm_profile_select_timeout_sec = float(merged.get("llm_profile_select_timeout_sec", 30.0))
|
llm_profile_select_timeout_sec = float(merged.get("llm_profile_select_timeout_sec", 30.0))
|
||||||
|
try:
|
||||||
|
ollama_num_predict = int(merged.get("ollama_num_predict", 512))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
ollama_num_predict = 512
|
||||||
|
|
||||||
return Settings(
|
return Settings(
|
||||||
# Database & Storage
|
# Database & Storage
|
||||||
@@ -774,6 +791,7 @@ def load_settings() -> Settings:
|
|||||||
llm_digest_timeout_sec=llm_digest_timeout_sec,
|
llm_digest_timeout_sec=llm_digest_timeout_sec,
|
||||||
llm_embedding_timeout_sec=llm_embedding_timeout_sec,
|
llm_embedding_timeout_sec=llm_embedding_timeout_sec,
|
||||||
llm_profile_select_timeout_sec=llm_profile_select_timeout_sec,
|
llm_profile_select_timeout_sec=llm_profile_select_timeout_sec,
|
||||||
|
ollama_num_predict=ollama_num_predict,
|
||||||
|
|
||||||
# Profiles & Behavior
|
# Profiles & Behavior
|
||||||
active_profiles=active_profiles,
|
active_profiles=active_profiles,
|
||||||
|
|||||||
@@ -2233,6 +2233,16 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
|||||||
has_tool_calls = " (has tool_calls)" if msg.get("tool_calls") else ""
|
has_tool_calls = " (has tool_calls)" if msg.get("tool_calls") else ""
|
||||||
debug_log(f" [{i}] {role}: {content}{has_tool_calls}", "planning")
|
debug_log(f" [{i}] {role}: {content}{has_tool_calls}", "planning")
|
||||||
|
|
||||||
|
# Bound worst-case generation latency: spoken answers are 1-2 sentences,
|
||||||
|
# so cap the chat model's output tokens. The default headroom sits well
|
||||||
|
# above this app's tool-call JSON, so capping never truncates a tool
|
||||||
|
# call. 0/negative disables it. See config.ollama_num_predict.
|
||||||
|
try:
|
||||||
|
_num_predict = int(getattr(cfg, 'ollama_num_predict', 0) or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
_num_predict = 0
|
||||||
|
_chat_extra_options = {"num_predict": _num_predict} if _num_predict > 0 else None
|
||||||
|
|
||||||
# Send messages to Ollama — try native tool calling first, fall back to text-based
|
# Send messages to Ollama — try native tool calling first, fall back to text-based
|
||||||
# if the model returns HTTP 400 (native tools API not supported).
|
# if the model returns HTTP 400 (native tools API not supported).
|
||||||
_dump_tools_schema = None if use_text_tools else tools_json_schema
|
_dump_tools_schema = None if use_text_tools else tools_json_schema
|
||||||
@@ -2242,7 +2252,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
|||||||
chat_model=cfg.ollama_chat_model,
|
chat_model=cfg.ollama_chat_model,
|
||||||
messages=messages,
|
messages=messages,
|
||||||
timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)),
|
timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)),
|
||||||
extra_options=None,
|
extra_options=_chat_extra_options,
|
||||||
tools=_dump_tools_schema,
|
tools=_dump_tools_schema,
|
||||||
thinking=getattr(cfg, 'llm_thinking_enabled', False),
|
thinking=getattr(cfg, 'llm_thinking_enabled', False),
|
||||||
)
|
)
|
||||||
@@ -2273,7 +2283,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
|||||||
chat_model=cfg.ollama_chat_model,
|
chat_model=cfg.ollama_chat_model,
|
||||||
messages=messages,
|
messages=messages,
|
||||||
timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)),
|
timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)),
|
||||||
extra_options=None,
|
extra_options=_chat_extra_options,
|
||||||
tools=None,
|
tools=None,
|
||||||
thinking=getattr(cfg, 'llm_thinking_enabled', False),
|
thinking=getattr(cfg, 'llm_thinking_enabled', False),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -287,6 +287,8 @@ Turn 4: LLM → {content: "Here's a comprehensive comparison of the iPhone 15 mo
|
|||||||
- `llm_tools_timeout_sec` (enrichment extraction)
|
- `llm_tools_timeout_sec` (enrichment extraction)
|
||||||
- `llm_embed_timeout_sec` (vector search)
|
- `llm_embed_timeout_sec` (vector search)
|
||||||
- `llm_chat_timeout_sec` (messages loop turn)
|
- `llm_chat_timeout_sec` (messages loop turn)
|
||||||
|
- Output bound:
|
||||||
|
- `ollama_num_predict` (default `512`, `0`/negative disables) caps the chat model's generated tokens per turn via the Ollama `num_predict` option on the messages-loop call. Spoken (TTS) answers are 1-2 sentences, so this never clips a normal answer; it bounds the worst-case latency of a model that occasionally rambles or loops. The default headroom sits well above this app's short tool-call JSON, so it does not truncate tool calls. Applied uniformly to the reply loop's chat call (both native-tools and text-tools paths); the small classification passes (intent judge, digests) keep their own caps. Note: this is a worst-case guard, not the primary latency lever, which is model size and GPU residency.
|
||||||
- Memory enrichment:
|
- Memory enrichment:
|
||||||
- `memory_enrichment_max_results` limits recalled snippets.
|
- `memory_enrichment_max_results` limits recalled snippets.
|
||||||
- `memory_digest_enabled` (default `null` = auto-on for SMALL models ≤7B, off for LARGE) distils the combined diary + graph dump into a short relevance-filtered note via a cheap LLM pass before injecting into the system prompt. See **Memory Digest for Small Models** below.
|
- `memory_digest_enabled` (default `null` = auto-on for SMALL models ≤7B, off for LARGE) distils the combined diary + graph dump into a short relevance-filtered note via a cheap LLM pass before injecting into the system prompt. See **Memory Digest for Small Models** below.
|
||||||
|
|||||||
@@ -30,8 +30,10 @@ class BrowseAndPlayTool(Tool):
|
|||||||
def description(self) -> str:
|
def description(self) -> str:
|
||||||
return (
|
return (
|
||||||
"Play a song / music video / clip on the shared screen by searching YouTube "
|
"Play a song / music video / clip on the shared screen by searching YouTube "
|
||||||
"and playing the first result. Use when the user asks you to play or watch "
|
"and playing a result. Use when the user asks you to play or watch "
|
||||||
"something. Only available in screen-share mode."
|
"something. Plays the first result by default; pass 'index' to play the "
|
||||||
|
"Nth result from the top of the search list (e.g. 'play the 3rd video' -> "
|
||||||
|
"index=3). Only available in screen-share mode."
|
||||||
)
|
)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
@@ -42,7 +44,16 @@ class BrowseAndPlayTool(Tool):
|
|||||||
"query": {
|
"query": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "What to play, e.g. 'IU Good Day' or 'lofi hip hop'.",
|
"description": "What to play, e.g. 'IU Good Day' or 'lofi hip hop'.",
|
||||||
}
|
},
|
||||||
|
"index": {
|
||||||
|
"type": "integer",
|
||||||
|
"description": (
|
||||||
|
"1-based position of the video to play in the search results, "
|
||||||
|
"counted from the top of the list. Defaults to 1 (first result). "
|
||||||
|
"Use for 'play the Nth video' / 'play the second one'."
|
||||||
|
),
|
||||||
|
"minimum": 1,
|
||||||
|
},
|
||||||
},
|
},
|
||||||
"required": ["query"],
|
"required": ["query"],
|
||||||
}
|
}
|
||||||
@@ -55,18 +66,25 @@ class BrowseAndPlayTool(Tool):
|
|||||||
reply_text="화면 공유 모드(STREAM_BROWSER=true)에서만 영상을 재생할 수 있습니다.",
|
reply_text="화면 공유 모드(STREAM_BROWSER=true)에서만 영상을 재생할 수 있습니다.",
|
||||||
)
|
)
|
||||||
query = ""
|
query = ""
|
||||||
|
index = 1
|
||||||
if args and isinstance(args, dict):
|
if args and isinstance(args, dict):
|
||||||
query = str(args.get("query", "")).strip()
|
query = str(args.get("query", "")).strip()
|
||||||
|
try:
|
||||||
|
index = int(args.get("index", 1) or 1)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
index = 1
|
||||||
|
if index < 1:
|
||||||
|
index = 1
|
||||||
if not query:
|
if not query:
|
||||||
return ToolExecutionResult(success=False, reply_text="재생할 내용을 알려주세요.")
|
return ToolExecutionResult(success=False, reply_text="재생할 내용을 알려주세요.")
|
||||||
if not _NODE_SCRIPT.exists():
|
if not _NODE_SCRIPT.exists():
|
||||||
return ToolExecutionResult(success=False, reply_text="브라우저 재생 도구를 찾을 수 없습니다.")
|
return ToolExecutionResult(success=False, reply_text="브라우저 재생 도구를 찾을 수 없습니다.")
|
||||||
|
|
||||||
context.user_print(f"▶️ 화면에서 '{query}' 재생 중…")
|
context.user_print(f"▶️ 화면에서 '{query}' 재생 중… (#{index})")
|
||||||
debug_log(f" ▶️ browseAndPlay '{query}'", "tools")
|
debug_log(f" ▶️ browseAndPlay '{query}' index={index}", "tools")
|
||||||
try:
|
try:
|
||||||
proc = subprocess.run(
|
proc = subprocess.run(
|
||||||
["node", str(_NODE_SCRIPT), query, "youtube"],
|
["node", str(_NODE_SCRIPT), query, "youtube", str(index)],
|
||||||
capture_output=True,
|
capture_output=True,
|
||||||
text=True,
|
text=True,
|
||||||
timeout=40,
|
timeout=40,
|
||||||
|
|||||||
@@ -6,16 +6,24 @@ video, or clip.
|
|||||||
|
|
||||||
### Behaviour
|
### Behaviour
|
||||||
|
|
||||||
- Public schema is a single required `query` string (what to play).
|
- Public schema is a required `query` string (what to play) plus an optional
|
||||||
|
`index` integer (1-based position in the search results, counted from the top
|
||||||
|
of the list). `index` defaults to `1` (first result), so existing callers and
|
||||||
|
"play X" requests are unchanged; "play the 3rd video" / "play the second one"
|
||||||
|
map to `index=3` / `index=2`.
|
||||||
- **Mode-gated**: only acts when `STREAM_BROWSER` is true (`cfg.stream_browser`).
|
- **Mode-gated**: only acts when `STREAM_BROWSER` is true (`cfg.stream_browser`).
|
||||||
In voice-only mode (false) there is no screen to show, so it returns a short
|
In voice-only mode (false) there is no screen to show, so it returns a short
|
||||||
message and does nothing.
|
message and does nothing.
|
||||||
- Drives the on-screen Chrome by subprocessing the Node CDP helper
|
- Drives the on-screen Chrome by subprocessing the Node CDP helper
|
||||||
`bot/scripts/stream-test/browse-search.mjs <query> youtube`, which searches
|
`bot/scripts/stream-test/browse-search.mjs <query> youtube <index>`, which
|
||||||
YouTube and plays the first result on display `:1`. The broadcast captures
|
searches YouTube and plays the chosen result on display `:1`. The broadcast
|
||||||
that display, so the playback is what viewers see.
|
captures that display, so the playback is what viewers see.
|
||||||
- Returns `success` with the played video's title, or a failure message if the
|
- The helper clicks the `index`-th `a#video-title` in the results list. The
|
||||||
helper/Chrome is unavailable. It does NOT make an LLM call.
|
index is clamped to the number of results actually returned, so asking for a
|
||||||
|
position beyond the list plays the last available result rather than failing.
|
||||||
|
- Returns `success` with the played video's title (and the resolved `index`), or
|
||||||
|
a failure message if the helper/Chrome is unavailable. It does NOT make an LLM
|
||||||
|
call.
|
||||||
|
|
||||||
### Principles
|
### Principles
|
||||||
|
|
||||||
|
|||||||
79
tests/test_browse_and_play_index.py
Normal file
79
tests/test_browse_and_play_index.py
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
"""Tests for browseAndPlay's ``index`` argument (play the Nth search result).
|
||||||
|
|
||||||
|
Behaviour verified:
|
||||||
|
- default plays the first result (index 1) and stays backward-compatible,
|
||||||
|
- an explicit index is forwarded to the Node helper as the 4th argv,
|
||||||
|
- bad / sub-1 index values clamp to 1,
|
||||||
|
- the index is advertised in the tool schema.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.jarvis.tools.builtin.browse_and_play import BrowseAndPlayTool, _NODE_SCRIPT
|
||||||
|
|
||||||
|
|
||||||
|
def _ctx():
|
||||||
|
cfg = Mock()
|
||||||
|
cfg.stream_browser = True
|
||||||
|
return Mock(cfg=cfg, user_print=Mock())
|
||||||
|
|
||||||
|
|
||||||
|
def _run(args):
|
||||||
|
tool = BrowseAndPlayTool()
|
||||||
|
with patch("src.jarvis.tools.builtin.browse_and_play.subprocess.run") as mock_run:
|
||||||
|
mock_run.return_value = Mock(
|
||||||
|
stdout=json.dumps({"ok": True, "title": "Some Video"}),
|
||||||
|
stderr="",
|
||||||
|
)
|
||||||
|
result = tool.run(args, _ctx())
|
||||||
|
return mock_run, result
|
||||||
|
|
||||||
|
|
||||||
|
def _argv(mock_run):
|
||||||
|
return list(mock_run.call_args[0][0])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.unit
|
||||||
|
def test_schema_exposes_index():
|
||||||
|
schema = BrowseAndPlayTool().inputSchema
|
||||||
|
assert "index" in schema["properties"]
|
||||||
|
assert schema["properties"]["index"]["type"] == "integer"
|
||||||
|
assert "query" in schema["required"]
|
||||||
|
assert "index" not in schema["required"] # optional
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.unit
|
||||||
|
def test_default_index_is_first():
|
||||||
|
mock_run, result = _run({"query": "IU Good Day"})
|
||||||
|
argv = _argv(mock_run)
|
||||||
|
assert argv[:4] == ["node", str(_NODE_SCRIPT), "IU Good Day", "youtube"]
|
||||||
|
assert argv[4] == "1"
|
||||||
|
assert result.success is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.unit
|
||||||
|
def test_explicit_index_forwarded():
|
||||||
|
mock_run, _ = _run({"query": "lofi", "index": 3})
|
||||||
|
assert _argv(mock_run)[4] == "3"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.unit
|
||||||
|
@pytest.mark.parametrize("bad", [0, -2, "nope", None])
|
||||||
|
def test_bad_index_clamps_to_one(bad):
|
||||||
|
mock_run, _ = _run({"query": "lofi", "index": bad})
|
||||||
|
assert _argv(mock_run)[4] == "1"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.unit
|
||||||
|
def test_voice_only_mode_does_not_play():
|
||||||
|
tool = BrowseAndPlayTool()
|
||||||
|
cfg = Mock()
|
||||||
|
cfg.stream_browser = False
|
||||||
|
ctx = Mock(cfg=cfg, user_print=Mock())
|
||||||
|
with patch("src.jarvis.tools.builtin.browse_and_play.subprocess.run") as mock_run:
|
||||||
|
result = tool.run({"query": "x", "index": 2}, ctx)
|
||||||
|
assert result.success is False
|
||||||
|
mock_run.assert_not_called()
|
||||||
112
tests/test_ollama_num_predict.py
Normal file
112
tests/test_ollama_num_predict.py
Normal file
@@ -0,0 +1,112 @@
|
|||||||
|
"""Tests for the ``ollama_num_predict`` chat-output cap.
|
||||||
|
|
||||||
|
The cap bounds worst-case reply latency by limiting how many tokens the chat
|
||||||
|
model may generate per turn. Spoken (TTS) answers are 1-2 sentences, so the
|
||||||
|
default headroom never clips a normal answer and stays above tool-call JSON.
|
||||||
|
|
||||||
|
These tests verify behaviour:
|
||||||
|
- the config default is present,
|
||||||
|
- the value is threaded into the Ollama request as the ``num_predict`` option,
|
||||||
|
- the reply loop forwards it to the chat call (and disables it at 0).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.jarvis.config import get_default_config
|
||||||
|
from src.jarvis.memory.conversation import DialogueMemory
|
||||||
|
from src.jarvis.reply.engine import run_reply_engine
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Config default
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_default_config_has_num_predict_cap():
|
||||||
|
config = get_default_config()
|
||||||
|
assert config["ollama_num_predict"] == 512
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Transport: extra_options.num_predict reaches the Ollama payload options
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@patch("jarvis.llm.requests.post")
|
||||||
|
def test_chat_with_messages_forwards_num_predict(mock_post):
|
||||||
|
from jarvis.llm import chat_with_messages
|
||||||
|
|
||||||
|
mock_resp = Mock()
|
||||||
|
mock_resp.status_code = 200
|
||||||
|
mock_resp.json.return_value = {"message": {"content": "ok"}}
|
||||||
|
mock_resp.raise_for_status = Mock()
|
||||||
|
mock_post.return_value = mock_resp
|
||||||
|
|
||||||
|
chat_with_messages(
|
||||||
|
"http://localhost:11434",
|
||||||
|
"test-large",
|
||||||
|
[{"role": "user", "content": "hi"}],
|
||||||
|
extra_options={"num_predict": 512},
|
||||||
|
)
|
||||||
|
_, kwargs = mock_post.call_args
|
||||||
|
options = (kwargs.get("json") or {}).get("options") or {}
|
||||||
|
assert options.get("num_predict") == 512
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Reply loop wiring
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _mock_cfg(num_predict):
|
||||||
|
cfg = Mock()
|
||||||
|
cfg.ollama_base_url = "http://localhost:11434"
|
||||||
|
cfg.ollama_chat_model = "test-large" # avoid SMALL-model text-tool path
|
||||||
|
cfg.ollama_num_predict = num_predict
|
||||||
|
cfg.voice_debug = False
|
||||||
|
cfg.llm_tools_timeout_sec = 8.0
|
||||||
|
cfg.llm_embed_timeout_sec = 10.0
|
||||||
|
cfg.llm_chat_timeout_sec = 45.0
|
||||||
|
cfg.llm_digest_timeout_sec = 8.0
|
||||||
|
cfg.memory_enrichment_max_results = 5
|
||||||
|
cfg.memory_enrichment_source = "diary"
|
||||||
|
cfg.memory_digest_enabled = False
|
||||||
|
cfg.tool_result_digest_enabled = False
|
||||||
|
cfg.location_ip_address = None
|
||||||
|
cfg.location_auto_detect = False
|
||||||
|
cfg.location_enabled = False
|
||||||
|
cfg.agentic_max_turns = 8
|
||||||
|
cfg.tool_search_max_calls = 3
|
||||||
|
cfg.tool_selection_strategy = "all"
|
||||||
|
cfg.tool_carryover_max_turns = 2
|
||||||
|
cfg.tool_carryover_per_entry_chars = 1200
|
||||||
|
cfg.mcps = {}
|
||||||
|
cfg.llm_thinking_enabled = False
|
||||||
|
cfg.tts_engine = "none"
|
||||||
|
cfg.ollama_embed_model = "test-embed"
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def _run_single_turn(cfg):
|
||||||
|
"""Drive one reply turn that answers in plain text and capture the
|
||||||
|
chat call's extra_options."""
|
||||||
|
with patch("src.jarvis.reply.engine.plan_query", return_value=[]), \
|
||||||
|
patch("src.jarvis.reply.engine.extract_search_params_for_memory", return_value={}), \
|
||||||
|
patch("src.jarvis.reply.engine.extract_text_from_response", return_value="Hello."), \
|
||||||
|
patch("src.jarvis.reply.engine.chat_with_messages") as mock_chat:
|
||||||
|
mock_chat.return_value = {"message": {"content": "Hello."}}
|
||||||
|
run_reply_engine(db=Mock(), cfg=cfg, tts=None,
|
||||||
|
text="hi", dialogue_memory=DialogueMemory())
|
||||||
|
assert mock_chat.called
|
||||||
|
return mock_chat.call_args.kwargs.get("extra_options")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.unit
|
||||||
|
def test_reply_loop_caps_output_when_enabled():
|
||||||
|
extra = _run_single_turn(_mock_cfg(512))
|
||||||
|
assert extra == {"num_predict": 512}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.unit
|
||||||
|
def test_reply_loop_no_cap_when_zero():
|
||||||
|
extra = _run_single_turn(_mock_cfg(0))
|
||||||
|
assert extra is None
|
||||||
@@ -110,6 +110,14 @@ class TestReplyLanguageDirective:
|
|||||||
directive = reply_language_directive("Korean", "melo")
|
directive = reply_language_directive("Korean", "melo")
|
||||||
assert directive is not None and "Korean" in directive
|
assert directive is not None and "Korean" in directive
|
||||||
|
|
||||||
|
def test_edge_is_multilingual(self):
|
||||||
|
# Edge TTS (the default Korean voice) is not English-only: no lock → the
|
||||||
|
# user's own language, and a lock is honoured (not forced to English).
|
||||||
|
assert reply_language_directive(None, "edge") is None
|
||||||
|
directive = reply_language_directive("Korean", "edge")
|
||||||
|
assert directive is not None and "Korean" in directive
|
||||||
|
assert directive != ENGLISH_ONLY_DIRECTIVE
|
||||||
|
|
||||||
|
|
||||||
class TestLoadAgentInstructions:
|
class TestLoadAgentInstructions:
|
||||||
"""Operator can extend the reply LLM's system prompt by dropping *.md files
|
"""Operator can extend the reply LLM's system prompt by dropping *.md files
|
||||||
|
|||||||
Reference in New Issue
Block a user