Compare commits

..

7 Commits

Author SHA1 Message Date
javis-bot
bb60c4db3b feat(tts): default to offline GPU MeloTTS with brain-role worker
Switch the default TTS engine from online Edge TTS to offline GPU
MeloTTS (privacy-first: reply text never leaves the host). Adds a
supervisord melo-worker program that runs in full/bot/brain roles and
synthesises Korean on the GPU (MELO_DEVICE=cuda, MELO_SPEED=1.5), a
baked /opt/melo py3.11 venv layer in the Dockerfile, and compose/env
defaults. Edge TTS stays available as an opt-in online fallback via
TTS_ENGINE=edge.

Verified on the RTX 5050 (Blackwell sm_120): melo-worker logs
"ready (lang=KR speed=1.5 device=cuda)", /health returns ok, and a
/synth round-trip returns a 177KB WAV.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-22 20:39:54 +09:00
javis-bot
8562548571 feat(deploy): add brain/app roles for GPU-split deployment
Run the GPU-heavy compute (Whisper STT, Ollama LLM, TTS) on a dedicated
brain host and the bot + browser on a separate no-GPU app host, talking
over the LAN.

- supervisord: new `brain` role (bridge only) and `app` role (bot +
  desktop/Chrome, no local bridge); `full`/`browser`/`bot` unchanged.
- compose: make BRIDGE_URL overridable (was hardcoded to loopback, which
  prevented the bot from reaching a remote bridge).
- docs: DEPLOY.md "GPU split" layout with per-host .env blocks; document
  the new roles in run-if-role.sh and .env.example.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-22 15:56:51 +09:00
javis-bot
db51bb92e6 feat(compose): allow external Ollama via OLLAMA_BASE_URL override
For the split where the LLM runs on a separate LAN host (e.g. a GPU box at
192.168.10.9) and the app stack runs elsewhere (192.168.10.5), the brain must
reach Ollama over the network. The javis service hardcoded
OLLAMA_BASE_URL=http://ollama:11434, ignoring any .env value, so the app could
only use the in-stack Ollama. Make it ${OLLAMA_BASE_URL:-http://ollama:11434}:
default unchanged (all-in-one), overridable to point at an external Ollama.
Document the external-Ollama setup in .env.example and DEPLOY.md, refresh the
browser-host IP examples (.9→.5), and de-hardcode the .9 example in novnc.ts.
2026-07-22 13:42:58 +09:00
javis-bot
ffc16665e5 fix(controlBrowser): never report moveMouse/search success without a real move
Some checks failed
Release / semantic-release (push) Successful in 19s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m15s
Release / build-linux (push) Failing after 7m7s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
moveMouse returned ok:true even when humanHover did nothing (no on-screen box)
or the selector never matched — recreating the "claims it moved but didn't"
bug. Now: humanHover returns a boolean (and brings the element into view first);
moveMouse returns ok:false when the target isn't found or has no on-screen box,
and when site=naver/... whose box isn't on the current page it navigates to the
site home first before hovering. search now reports input=human|api-fallback|api
so a silent fallback to cursor-less DOM input is visible, and the tool surfaces
that note in the reply instead of implying a human-like search happened.
2026-06-24 19:17:46 +09:00
javis-bot
5629da7e9f feat(controlBrowser): add moveMouse (hover) action for the visible cursor
The tool had no cursor-move/hover action, so "move the mouse to the search box"
had nothing to call and a weak model just claimed it had moved it. Add a
moveMouse action wired to human.humanHover (real xdotool cursor), targetable by
CSS selector or site= (that site's search box). Also clarify in the tool
description that 'navigate' types into the address bar (no mouse) while
'search'/'type' move the real cursor to the on-page box and click before
typing, so the model picks the visible-cursor path when the user wants it.
2026-06-24 19:14:04 +09:00
javis-bot
83999a5b0b fix(prompts): classify every sub-8B model (2b/4b/5b/6b) as SMALL
Some checks failed
Release / semantic-release (push) Successful in 24s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m16s
Release / build-linux (push) Failing after 7m21s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
detect_model_size only matched 1b/3b/7b, so a genuinely small model such as
qwen*:4b fell through to LARGE and got the terse, less-guided prompt set it
can't follow — contributing to off-tone, rambling replies. Extend the small
patterns to cover all sub-8B sizes (1b-7b) across :/-/_ separators and sync the
spec table.
2026-06-24 18:31:54 +09:00
javis-bot
d5fd218c86 fix(reply): stop weak models parroting persona example facts
A 4b model replied to "하이" with "테니스 연습을 Trenches Gym에서..." — it copied
the literal "box at Trenches Gym" few-shot example embedded in the persona
prompt and mangled boxing into tennis, presenting a prompt example as if it
were a real user fact. Remove the copyable proper-noun example and add an
explicit guard: use ONLY names/places/activities that literally appear in the
memory section, never borrow them from the instructions or example wording.
2026-06-24 18:31:54 +09:00
16 changed files with 352 additions and 66 deletions

View File

@@ -29,17 +29,27 @@ BRIDGE_PORT=8765
JARVIS_BRAIN_ENABLED=1
JARVIS_TTS_ENABLED=1
# faster-whisper device/compute. GPU by default (RTX 5050 / sm_120, verified).
# Falls back to CPU automatically if no GPU is passed to the container.
# On a NO-GPU host (e.g. an app host that offloads the LLM to a separate GPU box)
# set WHISPER_DEVICE=cpu AND WHISPER_COMPUTE_TYPE=int8 — float16 is GPU-only, so
# CPU needs int8 (STT is slower on CPU).
WHISPER_DEVICE=cuda
WHISPER_COMPUTE_TYPE=float16
# Optional explicit Piper voice model (.onnx). If empty, the jarvis default is used.
TTS_PIPER_MODEL_PATH=
# TTS engine: "edge" (default) uses Microsoft Edge TTS, a natural Korean neural
# voice. Set to "piper" for the offline English voice. NOTE: edge is ONLINE —
# reply text is sent to Microsoft's servers and needs internet.
TTS_ENGINE=edge
# Edge voice + speaking rate. Rate is a percentage (+45% ≈ 1.45×). Korean voices:
# ko-KR-HyunsuMultilingualNeural (M), ko-KR-InJoonNeural (M), ko-KR-SunHiNeural (F).
# TTS engine:
# melo (default) GPU MeloTTS — offline Korean voice, privacy-first (text never
# leaves the host), synthesises on the GPU via the melo-worker. Needs a
# GPU (MELO_DEVICE=cuda); falls back to CPU (slow) with MELO_DEVICE=cpu.
# edge Microsoft Edge TTS — natural Korean neural voice, but ONLINE: reply
# text is sent to Microsoft's servers and needs internet.
# piper offline English voice (fallback).
TTS_ENGINE=melo
# GPU MeloTTS knobs (used when TTS_ENGINE=melo).
MELO_DEVICE=cuda
MELO_SPEED=1.5
# Edge voice + speaking rate (used when TTS_ENGINE=edge). Rate is a percentage
# (+45% ≈ 1.45×). Korean voices: ko-KR-HyunsuMultilingualNeural (M),
# ko-KR-InJoonNeural (M), ko-KR-SunHiNeural (F).
EDGE_TTS_VOICE=ko-KR-HyunsuMultilingualNeural
EDGE_TTS_RATE=+45%
# Neural-only by default: if synthesis fails the bridge returns no audio rather
@@ -51,7 +61,13 @@ MELO_FALLBACK_PIPER=0
# Jarvis brain (Ollama-backed). In Docker these populate the rendered
# config (docker/jarvis-config.template.json). See src/jarvis/config.py.
# ---------------------------------------------------------------------------
# In docker-compose this is overridden to http://ollama:11434 automatically.
# In docker-compose this defaults to the in-stack `ollama` service
# (http://ollama:11434). To use an EXTERNAL Ollama on another LAN machine (e.g. a
# GPU host), set it here to that host's IP — the compose now respects this value:
# OLLAMA_BASE_URL=http://192.168.10.9:11434 # Ollama runs on the .9 host
# In that split setup: do NOT start the in-stack ollama/ollama-init (run
# `docker compose up -d javis`), pull the models on the LLM host, and make that
# host's Ollama listen on the LAN (OLLAMA_HOST=0.0.0.0:11434).
OLLAMA_BASE_URL=http://127.0.0.1:11434
# qwen2.5:3b — small non-reasoning instruct model. ~2.4GB, runs 100% on the GPU
# (the 8B offloads ~8% to CPU), warm voice turns ~2-4s vs ~5-7s on 8B. Clean
@@ -197,6 +213,10 @@ VOICE_SILENCE_MS=800
# full (default) everything in one container
# browser ONLY the desktop + Chrome + control-server (driven over the LAN)
# bot ONLY the bot + bridge + TTS (drives a REMOTE browser)
# brain ONLY the bridge (STT + reply + TTS) on a GPU host — no bot/desktop.
# The app host reaches it via BRIDGE_URL. See docs/DEPLOY.md "GPU split".
# app the bot + desktop/Chrome, but NOT the local bridge: offloads
# STT/LLM/TTS to a remote brain host (set BRIDGE_URL=http://<brain>:8765).
JARVIS_ROLE=full
# --- GPU per OS: pick the matching compose override via COMPOSE_FILE ---

View File

@@ -65,10 +65,19 @@ RUN ls -d /opt/venv/lib/python*/site-packages/nvidia/cublas/lib \
> /etc/ld.so.conf.d/nvidia-cu12.conf 2>/dev/null \
&& /sbin/ldconfig || true
# --- Korean voice: Microsoft Edge TTS (online neural). No model is baked — the
# `edge-tts` pip package (in requirements-bridge.txt) calls the MS service at
# runtime and the bridge transcodes the MP3 to PCM16 with ffmpeg. No heavy
# TTS build layer is needed. ---
# --- Korean voice: GPU MeloTTS (offline, privacy-first) in a separate /opt/melo
# py3.11 venv; see docker/setup-melo.sh. Heavy layer (torch cu128 GPU +
# transformers + MeCab + baked KR model), placed before the app COPY so it
# stays cached across source-only changes. Runtime device is MELO_DEVICE=cuda
# (compose), so TTS synthesises on the GPU next to Ollama + Whisper. Started
# only in full/bot/brain roles (see docker/supervisord.conf). The offline
# Edge TTS (`edge-tts`, in requirements-bridge.txt) remains available as an
# opt-in fallback via TTS_ENGINE=edge, but it is ONLINE (text leaves the
# host), so MeloTTS is the default. ---
COPY docker/setup-melo.sh /app/docker/setup-melo.sh
# Strip CR first: a Windows (autocrlf) checkout yields CRLF, which makes bash
# read `set -euxo pipefail\r` and abort. Idempotent on an LF tree.
RUN sed -i 's/\r$//' /app/docker/setup-melo.sh && bash /app/docker/setup-melo.sh
# --- Human input + window management for the on-screen Chrome control tool.
# xdotool injects real X pointer/keyboard events (visible cursor,

View File

@@ -17,6 +17,7 @@
// status | listTabs
// navigate {url} | back | forward | refresh
// newTab {url?} | closeTab {index?} | activateTab {index} | closePopups
// moveMouse {selector | site} (hover the real cursor, no click)
// click {selector} | type {text, selector?} | scroll {dir, notches?}
// pressKey {key} | screenshot {path}
import { chromium } from 'playwright';
@@ -40,6 +41,15 @@ if (!action) { out({ ok: false, error: 'no action' }); process.exit(1); }
const norm = (u) => (/^https?:\/\//i.test(u) ? u : `https://${u}`);
// Per-site homepage + search-box selector, shared by `search` and `moveMouse`.
const SITES = {
naver: { home: 'https://www.naver.com', box: '#query, input[name="query"]' },
google: { home: 'https://www.google.com', box: 'textarea[name="q"], input[name="q"]' },
daum: { home: 'https://www.daum.net', box: '#q, input[name="q"]' },
youtube: { home: 'https://www.youtube.com', box: 'input#search, input[name="search_query"]' },
bing: { home: 'https://www.bing.com', box: '#sb_form_q, input[name="q"]' },
};
// The genuinely-active tab is the one whose document is visible. Playwright has
// no "active page" accessor over CDP, so probe visibilityState (fixes treating
// tab 0 as active and breaking sequential ops on a specific tab).
@@ -103,13 +113,6 @@ try {
const q = String(cmd.query || '').trim();
if (!q) throw new Error('search: no query');
const siteKey = String(cmd.site || 'google').toLowerCase();
const SITES = {
naver: { home: 'https://www.naver.com', box: '#query, input[name="query"]' },
google: { home: 'https://www.google.com', box: 'textarea[name="q"], input[name="q"]' },
daum: { home: 'https://www.daum.net', box: '#q, input[name="q"]' },
youtube: { home: 'https://www.youtube.com', box: 'input#search, input[name="search_query"]' },
bing: { home: 'https://www.bing.com', box: '#sb_form_q, input[name="q"]' },
};
const s = SITES[siteKey] || SITES.google;
await front(page);
// 1) Go to the homepage.
@@ -122,23 +125,31 @@ try {
// 2) Click the on-page search box, type the query, submit.
const box = page.locator(s.box).first();
await box.waitFor({ state: 'visible', timeout: 15000 }).catch(() => {});
// Report which input path actually ran: 'human' = real xdotool cursor
// move + char typing; 'api-fallback' = the humanClick path threw and we
// fell back to cursor-less DOM click/fill; 'api' = no xdotool at all. This
// makes "did the cursor really move" verifiable from the result.
let searchInput;
if (HAS_XDOTOOL && cmd.human !== false) {
try {
await human.humanClick(page, box);
await human.humanType(q);
await human.pressKey('Return');
searchInput = 'human';
} catch {
searchInput = 'api-fallback';
await box.click().catch(() => {});
await box.fill(q).catch(() => {});
await page.keyboard.press('Enter').catch(() => {});
}
} else {
searchInput = 'api';
await box.click().catch(() => {});
await box.fill(q);
await page.keyboard.press('Enter');
}
await page.waitForLoadState('domcontentloaded').catch(() => {});
out({ ok: true, site: SITES[siteKey] ? siteKey : 'google', query: q, url: page.url(), title: await page.title().catch(() => '') });
out({ ok: true, site: SITES[siteKey] ? siteKey : 'google', query: q, url: page.url(), title: await page.title().catch(() => ''), input: searchInput });
break;
}
@@ -205,6 +216,45 @@ try {
break;
}
case 'moveMouse': {
// Move/hover the REAL cursor onto an element WITHOUT clicking. Target is a
// CSS selector, or site=naver/google/... for that site's search box.
// Only meaningful with xdotool (the visible cursor); with no xdotool there
// is no cursor to move, so report that rather than faking success. Every
// failure to actually move (no xdotool, selector never matches, element
// has no on-screen box) returns ok:false — we must never claim the cursor
// moved when it did not (the exact bug the user reported).
const siteKey = String(cmd.site || '').toLowerCase();
const selector = String(cmd.selector || '').trim() || (SITES[siteKey] ? SITES[siteKey].box : '');
if (!selector) throw new Error('moveMouse: no selector or known site');
if (!(HAS_XDOTOOL && cmd.human !== false)) {
out({ ok: false, error: 'no xdotool: cannot move the visible cursor on this host' });
break;
}
await front(page);
let locator = page.locator(selector).first();
let visible = await locator.waitFor({ state: 'visible', timeout: 8000 }).then(() => true).catch(() => false);
// A named site whose search box isn't on the current page: go to its home
// first (real omnibox), then target the box there.
if (!visible && SITES[siteKey]) {
try { await human.navigateOmnibox(SITES[siteKey].home); await page.waitForLoadState('domcontentloaded').catch(() => {}); }
catch { await page.goto(SITES[siteKey].home, { waitUntil: 'domcontentloaded' }).catch(() => {}); }
locator = page.locator(SITES[siteKey].box).first();
visible = await locator.waitFor({ state: 'visible', timeout: 8000 }).then(() => true).catch(() => false);
}
if (!visible) {
out({ ok: false, error: `moveMouse: target not found (${cmd.selector || siteKey})` });
break;
}
const moved = await human.humanHover(page, locator);
if (!moved) {
out({ ok: false, error: 'moveMouse: element has no on-screen box; cursor not moved' });
break;
}
out({ ok: true, target: cmd.selector || siteKey, input: 'human' });
break;
}
case 'click': {
const selector = String(cmd.selector || '').trim();
if (!selector) throw new Error('click: no selector');

View File

@@ -136,14 +136,18 @@ export async function navigateOmnibox(text) {
}
// Move the real cursor over an element (hover, no click) - e.g. to reveal a
// video player's controls or to focus it for a keyboard shortcut.
// video player's controls or to focus it for a keyboard shortcut. Returns true
// only if the element had an on-screen box and the cursor was actually moved;
// returns false when there is nothing to move to (so callers must not report
// success). Brings the element into view with a real wheel scroll first.
export async function humanHover(page, locator) {
const box = await locator.boundingBox().catch(() => null);
if (!box) return;
const box = await bringIntoView(page, locator);
if (!box) return false;
const g = await page.evaluate(() => ({ sx: window.screenX, sy: window.screenY, ow: window.outerWidth, oh: window.outerHeight, iw: window.innerWidth, ih: window.innerHeight }));
const bx = Math.max(0, Math.round((g.ow - g.iw) / 2));
const oy = g.sy + Math.max(0, g.oh - g.ih - bx);
await humanMove(Math.round(g.sx + bx + box.x + box.width * 0.5), Math.round(oy + box.y + box.height * 0.4));
return true;
}
export { sleep, rand };

View File

@@ -3,7 +3,8 @@
*
* Does not broadcast natively into Discord. Instead it shares a noVNC web URL
* that anyone can open in a browser to watch (and optionally control) the VNC
* desktop live. Set NOVNC_URL in .env (e.g. http://192.168.10.9:6080/vnc.html).
* desktop live. Set NOVNC_URL in .env to the browser host's LAN IP
* (e.g. http://<browser-host-ip>:6080/vnc.html).
*
* Stand up noVNC once on the host with websockify, e.g.:
* websockify --web=/usr/share/novnc 6080 localhost:5901
@@ -22,7 +23,7 @@ export class NoVncStreamer implements ScreenStreamer {
async start(_ctx: StreamContext): Promise<string> {
if (!this.config.novncUrl) {
return "NOVNC_URL이 설정되지 않았습니다 (.env). 예: http://192.168.10.9:6080/vnc.html";
return "NOVNC_URL이 설정되지 않았습니다 (.env). 예: http://<브라우저호스트IP>:6080/vnc.html";
}
this.active = true;
return `🖥️ VNC 화면 실시간 보기 (브라우저): ${this.config.novncUrl}`;

View File

@@ -66,8 +66,13 @@ services:
- path: .env
required: false
environment:
# Point the brain at the ollama service and the bot at the in-container bridge.
OLLAMA_BASE_URL: http://ollama:11434
# Point the brain at Ollama. Default is the in-stack `ollama` service
# (all-in-one). Override OLLAMA_BASE_URL in .env to reach an EXTERNAL Ollama
# over the LAN (e.g. a GPU host at http://192.168.10.9:11434) — used when
# this stack runs on a separate machine from the LLM. When overridden, do
# NOT start the in-stack `ollama`/`ollama-init` services (bring the app up
# with `docker compose up -d javis`) and pull the models on the LLM host.
OLLAMA_BASE_URL: ${OLLAMA_BASE_URL:-http://ollama:11434}
OLLAMA_CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen2.5:3b}
# Auxiliary small-model calls (intent judge, tool router, arg extraction,
# query decomposition) run on this fast model so the big chat model only
@@ -82,11 +87,17 @@ services:
WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda}
WHISPER_COMPUTE_TYPE: ${WHISPER_COMPUTE_TYPE:-float16}
# TTS engine. Rendered into /app/config/jarvis.json via envsubst (the
# bridge reads that JSON BEFORE the env, so it must carry the real engine,
# not a hardcoded one — otherwise Korean text is read by the English Piper
# voice). Default edge; .env can override (e.g. piper for offline).
TTS_ENGINE: ${TTS_ENGINE:-edge}
# Edge TTS voice + rate (the chosen natural Korean voice). NOTE: edge is an
# bridge reads that JSON BEFORE the env, so it must carry the real engine).
# Default melo: GPU MeloTTS (offline, privacy-first) via the melo-worker.
# Override with TTS_ENGINE=edge for the online Microsoft neural voice (text
# leaves the host) or piper for the offline English voice.
TTS_ENGINE: ${TTS_ENGINE:-melo}
# GPU MeloTTS worker knobs. MELO_DEVICE=cuda runs synth on the GPU (next to
# Ollama + Whisper); cpu is the no-GPU fallback. MELO_SPEED must always be
# set — supervisord's melo-worker expands %(ENV_MELO_SPEED)s at startup.
MELO_DEVICE: ${MELO_DEVICE:-cuda}
MELO_SPEED: ${MELO_SPEED:-1.5}
# Edge TTS voice + rate (used only when TTS_ENGINE=edge). NOTE: edge is an
# ONLINE engine — reply text is sent to Microsoft and needs internet.
EDGE_TTS_VOICE: ${EDGE_TTS_VOICE:-ko-KR-HyunsuMultilingualNeural}
EDGE_TTS_RATE: ${EDGE_TTS_RATE:-+45%}
@@ -101,10 +112,17 @@ services:
# on short/noisy Discord speech. Lower to 1 for minimum latency.
STT_BEAM_SIZE: ${STT_BEAM_SIZE:-5}
VOICE_SILENCE_MS: ${VOICE_SILENCE_MS:-600}
BRIDGE_URL: http://127.0.0.1:8765
# Split-deployment role: full (default, all-in-one), browser (only the
# desktop + Chrome + CDP, reused over the LAN), or bot (only bot + bridge
# + TTS, driving a remote browser via CDP_HOST). See docker/run-if-role.sh.
# Where the bot reaches the brain bridge (STT + reply + TTS). Loopback for
# all-in-one. In a GPU-split layout the bot host (JARVIS_ROLE=app) points
# this at the brain host: BRIDGE_URL=http://192.168.10.9:8765 (that host
# runs JARVIS_ROLE=brain with BRIDGE_HOST=0.0.0.0 + SETTINGS_PUBLISH_BIND=0.0.0.0).
BRIDGE_URL: ${BRIDGE_URL:-http://127.0.0.1:8765}
# Split-deployment role (see docker/run-if-role.sh):
# full (default) everything in one container
# browser only desktop + Chrome + control-server (driven over the LAN)
# bot only bot + bridge + TTS (drives a remote browser)
# brain only the bridge (STT + reply + TTS) — GPU host, no bot/desktop
# app bot + browser/desktop, NO local bridge (uses a remote brain)
JARVIS_ROLE: ${JARVIS_ROLE:-full}
# Chrome CDP bind address INSIDE the container. 0.0.0.0 lets a remote bot
# (JARVIS_ROLE=bot on another PC) drive this host's browser. Loopback by
@@ -112,7 +130,7 @@ services:
CDP_BIND: ${CDP_BIND:-127.0.0.1}
CDP_PORT: ${CDP_PORT:-9222}
# Where the bot drives Chrome. Loopback for full/browser; on a remote bot
# set CDP_HOST to the browser host's LAN IP (e.g. 192.168.10.9).
# set CDP_HOST to the browser host's LAN IP (e.g. 192.168.10.5).
CDP_HOST: ${CDP_HOST:-127.0.0.1}
# Browser-control endpoint. The browser host serves it (BIND/PORT); a
# remote bot sets BROWSER_CONTROL_URL=http://<browser-host>:8777 so its

View File

@@ -11,6 +11,10 @@
# JARVIS_ROLE=browser only the desktop + Chrome + CDP (reused over the LAN)
# JARVIS_ROLE=bot only the bot + bridge + TTS (drives a remote browser
# via CDP_HOST/CDP_PORT)
# JARVIS_ROLE=brain only the bridge (STT + reply + TTS) on a GPU host; no
# bot, no desktop. The app host reaches it via BRIDGE_URL.
# JARVIS_ROLE=app the bot + desktop/Chrome, but NOT the local bridge:
# offloads STT/LLM/TTS to a remote brain host (BRIDGE_URL)
set -e
want="$1"; shift
role="${JARVIS_ROLE:-full}"

View File

@@ -14,7 +14,7 @@ serverurl=unix:///run/supervisor.sock
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:xvnc]
command=/app/docker/run-if-role.sh full,browser /app/docker/run-xvnc.sh
command=/app/docker/run-if-role.sh full,browser,app /app/docker/run-xvnc.sh
priority=100
autorestart=true
stdout_logfile=/dev/stdout
@@ -23,7 +23,7 @@ stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:pulse]
command=/app/docker/run-if-role.sh full,browser /app/docker/run-pulse.sh
command=/app/docker/run-if-role.sh full,browser,app /app/docker/run-pulse.sh
priority=150
autorestart=true
stdout_logfile=/dev/stdout
@@ -32,7 +32,7 @@ stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:xfce]
command=/app/docker/run-if-role.sh full,browser /app/docker/run-xfce.sh
command=/app/docker/run-if-role.sh full,browser,app /app/docker/run-xfce.sh
priority=200
autorestart=true
stdout_logfile=/dev/stdout
@@ -41,7 +41,7 @@ stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:novnc]
command=/app/docker/run-if-role.sh full,browser websockify --web=/usr/share/novnc 6080 localhost:5901
command=/app/docker/run-if-role.sh full,browser,app websockify --web=/usr/share/novnc 6080 localhost:5901
priority=250
autorestart=true
stdout_logfile=/dev/stdout
@@ -49,11 +49,30 @@ stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
# (No TTS worker program: the default Edge TTS engine synthesises in-process in
# the bridge via the `edge-tts` package — no warm model/worker is needed.)
[program:melo-worker]
; Warm GPU MeloTTS Korean voice (offline) in its own py3.11 venv (/opt/melo). The
; bridge's synthesize() POSTs here when TTS_ENGINE=melo. Runs wherever the bridge
; runs (full/bot/brain) so a GPU brain host serves TTS on the GPU. If TTS_ENGINE
; =edge the bridge synthesises in-process and never calls this worker.
command=/app/docker/run-if-role.sh full,bot,brain /opt/melo/bin/python /app/bridge/melo_worker.py
directory=/app
; HF_HOME points at the dedicated, image-baked melo cache (warmed in
; setup-melo.sh). The whisper_cache volume is mounted over /root/.cache/huggingface,
; so without this the pre-cached BERT + KR checkpoint would be shadowed and
; re-downloaded (and would fail offline). HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE
; force pure-cache reads. MELO_DEVICE/MELO_SPEED inherit from the container env
; (compose sets both with defaults cuda / 1.5), so MELO_SPEED must always be set
; in the env or this %(ENV_x)s expansion fails at startup (compose guarantees it).
environment=MELO_LANGUAGE="KR",MELO_SPEED="%(ENV_MELO_SPEED)s",MELO_DEVICE="%(ENV_MELO_DEVICE)s",MELO_WORKER_HOST="127.0.0.1",MELO_WORKER_PORT="8770",HF_HOME="/opt/melo-cache",HF_HUB_OFFLINE="1",TRANSFORMERS_OFFLINE="1"
priority=280
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:bridge]
command=/app/docker/run-if-role.sh full,bot /opt/venv/bin/python -m bridge.server
command=/app/docker/run-if-role.sh full,bot,brain /opt/venv/bin/python -m bridge.server
directory=/app
priority=300
autorestart=true
@@ -63,7 +82,7 @@ stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:chrome]
command=/app/docker/run-if-role.sh full,browser /app/docker/run-chrome.sh
command=/app/docker/run-if-role.sh full,browser,app /app/docker/run-chrome.sh
priority=350
autorestart=true
stdout_logfile=/dev/stdout
@@ -75,7 +94,7 @@ stderr_logfile_maxbytes=0
; Browser-control HTTP endpoint on the BROWSER HOST. A remote `bot` posts
; commands here so xdotool / CDP run on THIS machine (real input on this
; screen). Only meaningful in full/browser roles. Internal network only.
command=/app/docker/run-if-role.sh full,browser node /app/bot/scripts/stream-test/control-server.mjs
command=/app/docker/run-if-role.sh full,browser,app node /app/bot/scripts/stream-test/control-server.mjs
directory=/app/bot
priority=360
autorestart=true
@@ -85,7 +104,7 @@ stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:bot]
command=/app/docker/run-if-role.sh full,bot /app/docker/run-bot.sh
command=/app/docker/run-if-role.sh full,bot,app /app/docker/run-bot.sh
directory=/app/bot
priority=400
autorestart=true

View File

@@ -30,7 +30,7 @@ The on-screen Chrome, real mouse/keyboard (xdotool) and screen live on the
**browser host**. Your PC runs the **bot** and drives that browser over the
internal network — no auth (internal only).
### Browser host (the LAN machine that shows Chrome, e.g. 192.168.10.9)
### Browser host (the LAN machine that shows Chrome, e.g. 192.168.10.5)
```
# .env
@@ -50,7 +50,7 @@ Watch it on this machine’s VNC (`localhost:5901`) / noVNC (`localhost:6080`).
```
# .env
JARVIS_ROLE=bot
BROWSER_CONTROL_URL=http://192.168.10.9:8777 # the browser host's LAN IP
BROWSER_CONTROL_URL=http://192.168.10.5:8777 # the browser host's LAN IP
COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml # Ubuntu/macOS (":" )
# COMPOSE_FILE=docker-compose.yml;docker-compose.gpu-windows.yml # Windows 11 (";" )
DISCORD_SELFBOT_TOKEN=...
@@ -63,6 +63,73 @@ The bot’s `controlBrowser` tool posts commands to `BROWSER_CONTROL_URL`, so
"네이버에서 X 검색", "구글로 돌아가" etc. drive the **browser host’s** Chrome with real
human-style input (visible on its VNC).
### External Ollama (LLM on a separate host)
To run Ollama on a different LAN machine (e.g. a dedicated GPU host at
`192.168.10.9`) instead of inside this stack:
```
# .env on the app host
OLLAMA_BASE_URL=http://192.168.10.9:11434
```
Then start the app WITHOUT the in-stack LLM services: `docker compose up -d javis`
(do not `docker compose up -d`, which would also start `ollama`/`ollama-init`).
On the LLM host, make Ollama listen on the LAN (`OLLAMA_HOST=0.0.0.0:11434`) and
pull the models there (`ollama pull <chat>`, `<intent>`, `<embed>`). The compose
respects `OLLAMA_BASE_URL`, defaulting to the in-stack `ollama` service when unset.
## C. GPU split: brain host (.9, GPU) + app host (.5, no GPU)
All the GPU-heavy compute (Whisper STT, the Ollama LLM, and TTS) runs on the
**brain host** (a GPU box, e.g. `192.168.10.9`). The **app host** (no GPU, e.g.
`192.168.10.5`) runs the Discord bot and the on-screen browser, and offloads
every STT/LLM/TTS call to the brain over the LAN.
Flow: the bot (app host) POSTs captured voice to the brain's bridge; the brain
runs STT -> LLM -> TTS on the GPU and returns the reply audio; when the LLM
drives the browser, the brain's `controlBrowser` tool POSTs to the app host's
control-server (port 8777), so Chrome moves on the app host's screen.
### Brain host (.9 — GPU: STT + LLM + TTS)
```
# .env
JARVIS_ROLE=brain
BRIDGE_HOST=0.0.0.0 # bridge listens on the LAN
SETTINGS_PUBLISH_BIND=0.0.0.0 # publish bridge port 8765 to the LAN
OLLAMA_BASE_URL=http://192.168.10.9:11434 # Ollama on this host (bind 0.0.0.0)
OLLAMA_CHAT_MODEL=qwen2.5:7b
WHISPER_DEVICE=cuda # STT on the GPU
WHISPER_COMPUTE_TYPE=float16
TTS_ENGINE=melo # GPU MeloTTS (offline, privacy-first)
MELO_DEVICE=cuda # TTS on the GPU (melo-worker)
BROWSER_CONTROL_URL=http://192.168.10.5:8777 # drive the app host's Chrome
COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml # GPU (":" on Linux)
# STT (Whisper), LLM (Ollama) and TTS (MeloTTS) all run on this host's GPU.
# Ollama listens on 0.0.0.0:11434 (systemd or a container).
docker compose up -d javis # brain bridge + melo-worker only (no bot, no desktop)
```
### App host (.5 — bot + browser, no GPU)
```
# .env
JARVIS_ROLE=app
BRIDGE_URL=http://192.168.10.9:8765 # offload STT/LLM/TTS to the brain
CDP_PUBLISH_BIND=0.0.0.0 # publish control-server 8777 to the LAN
STREAM_HW=0 # no NVENC on a no-GPU host
DISCORD_SELFBOT_TOKEN=...
DISCORD_GUILD_ID=...
# no GPU → leave COMPOSE_FILE unset (base compose only)
docker compose up -d javis # bot + desktop + Chrome + control-server
```
The `brain` and `app` roles are backward-compatible additions to `full`/`browser`
/`bot`; see `docker/run-if-role.sh`.
## Windows 11 notes
- Install the NVIDIA driver on Windows and enable GPU in Docker Desktop

View File

@@ -22,11 +22,15 @@ class ModelSize(Enum):
LARGE = "large" # 8b+ - can infer tool usage from context
# Model size patterns - models matching these are considered SMALL
# Model size patterns - models matching these are considered SMALL.
# Covers every sub-8B size (1b-7b): these models need the explicit, repeated
# tool/greeting/instruction constraints and falter on the terse LARGE prompt.
# Without 2b/4b/5b/6b here a genuinely small model (e.g. qwen*:4b) was
# misclassified as LARGE and given the less-guided prompt set.
_SMALL_MODEL_PATTERNS = (
":1b", ":3b", ":7b",
"-1b", "-3b", "-7b",
"_1b", "_3b", "_7b",
":1b", ":2b", ":3b", ":4b", ":5b", ":6b", ":7b",
"-1b", "-2b", "-3b", "-4b", "-5b", "-6b", "-7b",
"_1b", "_2b", "_3b", "_4b", "_5b", "_6b", "_7b",
"gemma4", # Gemma 4 - always small regardless of tag
)

View File

@@ -4,7 +4,7 @@ This module provides model-size-aware prompt generation for the reply engine.
### Problem Statement
Small models (1b, 3b, 7b parameters) lack the reasoning capacity to infer when NOT to use tools. When given prompts like "Proactively use available tools," they may incorrectly call tools for simple greetings like "hello" or "ni hao" because they cannot distinguish between:
Small models (every sub-8B size, 1b-7b parameters) lack the reasoning capacity to infer when NOT to use tools. When given prompts like "Proactively use available tools," they may incorrectly call tools for simple greetings like "hello" or "ni hao" because they cannot distinguish between:
- Requests that require tools (weather, search, data retrieval)
- Simple conversation (greetings, small talk, general knowledge)
@@ -14,7 +14,7 @@ The module detects model size from the model name and selects appropriate prompt
| Model Size | Detection Pattern | Tool Prompts |
|------------|-------------------|--------------|
| SMALL | `:1b`, `:3b`, `:7b`, `gemma4` | Conservative — explicit "DO NOT use tools for greetings" + worked negative examples + repetition |
| SMALL | `:1b`-`:7b` (every size 1-7B, all separators), `gemma4` | Conservative — explicit "DO NOT use tools for greetings" + worked negative examples + repetition |
| LARGE | All others (8b+) | Proactive — "use tools confidently" + short anti-confabulation + auto-derive clause |
### Architecture

View File

@@ -120,10 +120,13 @@ _SYSTEM_PROMPT_TEMPLATE: str = (
"'tell me a joke', 'chat with me'), never reply with a bare greeting like 'Hey there!', "
"'Hi!', 'How can I help you?', or a generic observation about an unrelated topic. "
"When the 'Information the user has shared…' section is present, you MUST pick one concrete "
"fact from it and build the reply around that fact (e.g. 'You mentioned you box at Trenches "
"Gym — how's training going this week?'). Do not talk about things that are not in that "
"section. Only when that section is absent may you invent a fresh observation, question, or "
"joke. Produce a varied response each time — do not repeat a previous reply verbatim. "
"fact from it and build the reply around that fact, opening with a short natural reference to "
"it. CRITICAL: use ONLY names, places, activities, and details that literally appear in that "
"section — never borrow any name, place, or activity from these instructions or from any "
"example wording, and never invent specifics that are not in that section. Do not talk about "
"things that are not in that section. Only when that section is absent may you invent a fresh "
"observation, question, or joke. Produce a varied response each time — do not repeat a "
"previous reply verbatim. "
"Banned phrasings: 'I can only tell you what you have shared with me in this conversation', "
"'I don't have access to any personal information outside of what you tell me', 'I don't have "
"personal details outside of our conversation history', 'I do not store personal details "

View File

@@ -45,9 +45,16 @@ class ControlBrowserTool(Tool):
"Use this (NOT webSearch) whenever the user wants something done or shown IN the "
"browser on screen: open a website or URL, search on a specific site (action "
"'search' with site=naver/google/daum/youtube/bing), go back/forward, refresh, "
"manage tabs (list/new/close/switch), close popups, click, type, scroll, or "
"screenshot. webSearch only returns text and shows nothing on screen; this tool "
"actually navigates the visible browser. Only available in screen-share mode. "
"manage tabs (list/new/close/switch), close popups, move the mouse onto an element, "
"click, type, scroll, or screenshot. webSearch only returns text and shows nothing "
"on screen; this tool actually navigates the visible browser. "
"Cursor behaviour matters: 'search' and 'type' (with a selector) move the REAL mouse "
"cursor to the on-page box, click it, then type one character at a time — use these "
"when the user wants to search or type on a page. 'navigate' only types the URL into "
"the address bar (no mouse movement). 'moveMouse' moves/hovers the visible cursor "
"onto an element (selector, or site=naver/... for that site's search box) WITHOUT "
"clicking — use it when the user just asks to move the mouse somewhere. "
"Only available in screen-share mode. "
"Never claim you did any of this unless this tool returns success."
)
@@ -61,16 +68,16 @@ class ControlBrowserTool(Tool):
"enum": [
"status", "listTabs", "navigate", "search", "back",
"forward", "refresh", "newTab", "closeTab", "activateTab",
"closePopups", "click", "type", "scroll", "pressKey",
"closePopups", "moveMouse", "click", "type", "scroll", "pressKey",
"screenshot",
],
"description": "What to do in the browser.",
},
"url": {"type": "string", "description": "Target URL/site for navigate/newTab (e.g. 'naver.com')."},
"query": {"type": "string", "description": "Search text for action 'search'."},
"site": {"type": "string", "description": "Search site for action 'search': naver, google, daum, youtube, bing."},
"site": {"type": "string", "description": "Site for action 'search' or 'moveMouse': naver, google, daum, youtube, bing. For moveMouse it targets that site's search box."},
"index": {"type": "integer", "description": "Tab index for closeTab/activateTab (from listTabs)."},
"selector": {"type": "string", "description": "CSS selector for click/type."},
"selector": {"type": "string", "description": "CSS selector for click/type/moveMouse."},
"text": {"type": "string", "description": "Text to type."},
"key": {"type": "string", "description": "Key to press, e.g. 'Return', 'Escape'."},
"dir": {"type": "string", "description": "Scroll direction: 'down' or 'up'."},
@@ -149,7 +156,12 @@ class ControlBrowserTool(Tool):
if action == "navigate":
return f"브라우저에서 {data.get('url', args.get('url'))} 로 이동했습니다."
if action == "search":
return f"{data.get('site', '')}에서 '{data.get('query', args.get('query'))}'를 검색해 화면에 띄웠습니다."
base = f"{data.get('site', '')}에서 '{data.get('query', args.get('query'))}'를 검색해 화면에 띄웠습니다."
# Flag when the real cursor path didn't run, so a silent fallback to
# cursor-less DOM input is visible rather than reported as "human".
if data.get("input") in ("api", "api-fallback"):
base += " (참고: 실제 마우스 커서 이동 없이 처리됨)"
return base
if action in ("back", "forward", "refresh"):
return f"브라우저: {action} 완료 ({data.get('url', '')})."
if action in ("status", "listTabs"):
@@ -164,6 +176,9 @@ class ControlBrowserTool(Tool):
return f"탭 {data.get('active')}번으로 전환했습니다."
if action == "closePopups":
return f"팝업/빈 탭 {data.get('closed')}개를 닫았습니다."
if action == "moveMouse":
target = data.get("target") or args.get("selector") or args.get("site") or "대상"
return f"마우스 커서를 {target} 위치로 옮겼습니다."
if action == "screenshot":
return f"화면을 캡처했습니다: {data.get('path')}"
return "완료했습니다."

View File

@@ -0,0 +1,56 @@
"""Tests for the controlBrowser tool's action surface.
These are deterministic schema/summary checks — they do not drive a real
browser. The actual cursor movement is exercised live on the browser host
(xdotool + CDP), which these tests cannot reach.
"""
import pytest
from jarvis.tools.builtin.control_browser import ControlBrowserTool
@pytest.fixture
def tool():
return ControlBrowserTool()
def test_movemouse_is_an_exposed_action(tool):
# A weak model confabulated "moved the mouse" because no move/hover action
# existed to call. The cursor-move capability must be a real action so the
# request "move the mouse to the search box" maps to a tool call.
enum = tool.inputSchema["properties"]["action"]["enum"]
assert "moveMouse" in enum
def test_movemouse_summary_reports_the_target(tool):
summary = tool._summarise("moveMouse", {"site": "naver"}, {"ok": True, "target": "naver"})
assert "마우스" in summary and "naver" in summary
def test_description_distinguishes_cursor_paths(tool):
# The model must know navigate is address-bar only (no mouse) while
# search/type/moveMouse move the real cursor — that distinction is the
# whole point of the fix.
desc = tool.description
assert "moveMouse" in desc
assert "address bar" in desc # navigate is described as address-bar typing
def test_search_summary_flags_cursorless_fallback(tool):
# When the real xdotool cursor path didn't run, the summary must say so
# rather than implying a human-like search happened.
human = tool._summarise("search", {"query": "날씨"}, {"ok": True, "site": "naver", "query": "날씨", "input": "human"})
assert "참고: 실제 마우스" not in human
fell_back = tool._summarise("search", {"query": "날씨"}, {"ok": True, "site": "naver", "query": "날씨", "input": "api-fallback"})
assert "실제 마우스 커서 이동 없이" in fell_back
def test_movemouse_summary_only_runs_on_success(tool):
# _summarise is only called on ok:true; an ok:false (target not found / no
# xdotool) is handled by run() as a failure reply, so a failed move can no
# longer be reported as "moved". Sanity-check the success summary names a
# target rather than a placeholder when one is present.
summary = tool._summarise("moveMouse", {"selector": "#query"}, {"ok": True, "target": "#query"})
assert "#query" in summary

View File

@@ -21,9 +21,15 @@ class TestModelSizeDetection:
("gemma:7b", True),
("phi3:3b", True),
("qwen2:7b", True),
# Sub-8B sizes that were previously misclassified as LARGE.
("qwen3.5:4b", True), # the deployed model that produced weak, off-tone replies
("gemma2:2b", True),
("model:5b", True),
("model:6b", True),
# Various separators
("model-3b-instruct", True),
("model_1b_chat", True),
("model-4b-instruct", True),
# Large models (should return LARGE)
("gpt-oss:20b", False),
("llama3.1:8b", False),

View File

@@ -60,6 +60,16 @@ class TestBuildSystemPrompt:
assert "SINGLE sentence" in prompt
assert "never add it as " in prompt
def test_persona_has_no_copyable_proper_noun_examples(self):
# A weak model parroted the literal "Trenches Gym" example from the
# persona as if it were a real user fact (boxing mangled to tennis).
# The persona must not embed copyable personal proper nouns, and must
# tell the model to use ONLY facts that literally appear in the memory
# section — never borrow names/places from the instructions themselves.
prompt = build_system_prompt("Jarvis")
assert "Trenches" not in prompt
assert "never borrow any name, place, or activity from these instructions" in prompt
class TestOutputLanguageDirective:
"""A deployment may lock replies to a single language via OUTPUT_LANGUAGE.