16 Commits

Author SHA1 Message Date
javis-bot
086dd5cde7 fix: accept edge as a valid tts_engine and migrate stale persisted engines
Some checks failed
Release / semantic-release (push) Successful in 34s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Release / build-linux (push) Failing after 7m9s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
load_settings() coerced any tts_engine outside {piper, chatterbox} to piper, so
with TTS_ENGINE=edge the reply engine saw "piper" and treated the voice as
English-only in reply_language_directive() (only the OUTPUT_LANGUAGE lock kept
replies Korean). Add "edge" (and "melo") to the accepted set so the engine is
labelled multilingual correctly.

Also: a stale tts_engine in the persistent /data/jarvis-settings.json (melo/xtts
from an earlier voice, no longer built) would override the configured engine via
the entrypoint merge and leave the bot silent. Reset those to the env engine
during the merge.

Verified: load_settings() with tts_engine=edge now returns "edge"; the merge
maps melo/xtts -> edge; reply_language_directive("edge") is multilingual; 27
tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 03:49:53 +09:00
javis-bot
f64d76e737 feat: use Edge TTS (Korean Hyunsu voice @ +45%) as the default voice
Some checks failed
Release / semantic-release (push) Successful in 31s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled
The user chose Microsoft Edge TTS, voice ko-KR-HyunsuMultilingualNeural at rate
+45% (~1.45x), as the natural Korean voice. Wire it into the bridge and make it
the default engine.

- bridge/server.py: _edge_synthesize() calls edge-tts and transcodes the MP3 to
  PCM16 mono WAV with the system ffmpeg (temp file for a correct header);
  TTS_ENGINE default -> edge; EDGE_TTS_VOICE / EDGE_TTS_RATE env-driven
- requirements-bridge.txt: add edge-tts (lightweight; httpx)
- compose/.env.example/README: TTS_ENGINE=edge + EDGE_TTS_* knobs; note the
  online/privacy trade-off (reply text is sent to Microsoft, needs internet)
- drop the now-unused MeloTTS build layer (Dockerfile) and melo-worker
  (supervisord) — edge synthesises in-process, no model/worker baked, slimmer
  and faster image; settings UI engine list -> edge/piper, restart only bridge

Verified on host: edge-tts -> ffmpeg yields a valid 16-bit mono 24kHz WAV;
envsubst renders tts_engine=edge; docker build --check + 26 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 03:44:15 +09:00
javis-bot
11c3621093 fix: make container TTS engine env-driven so melo isn't overridden by piper
Some checks failed
Release / semantic-release (push) Successful in 30s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Release / build-linux (push) Failing after 7m3s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
docker/jarvis-config.template.json hardcoded "tts_engine": "piper". entrypoint
renders it into /app/config/jarvis.json, and bridge _tts_engine_setting() reads
that JSON BEFORE the env — so TTS_ENGINE=melo in .env was ignored and the bot
synthesised Korean with the English Piper voice (the "foreign accent" the user
heard); the warm melo-worker sat unused.

Template now carries ${TTS_ENGINE}; compose sets TTS_ENGINE=${TTS_ENGINE:-melo}
so envsubst renders the real engine. Verified: envsubst with TTS_ENGINE=melo
yields "tts_engine": "melo", and `docker compose config` passes TTS_ENGINE=melo.
Added a regression test that the template stays env-driven and renders the
configured engine.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 03:27:33 +09:00
javis-bot
7ad5d99380 Revert "feat: replace MeloTTS with Coqui XTTS-v2 natural Korean voice"
Some checks failed
Release / semantic-release (push) Successful in 35s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m16s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Release / build-linux (push) Has been cancelled
This reverts commit 39a0944105.
2026-06-23 03:15:54 +09:00
javis-bot
39a0944105 feat: replace MeloTTS with Coqui XTTS-v2 natural Korean voice
Some checks failed
Release / semantic-release (push) Successful in 30s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Release / build-linux (push) Has been cancelled
MeloTTS's single Korean speaker sounded non-native ("foreign accent"). Swap it
for Coqui XTTS-v2 with the built-in female studio speaker "Ana Florence"
(language ko), the natural voice used in earlier local runs.

- bridge/xtts_worker.py: new warm HTTP worker (own /opt/xtts venv), same
  /synth + /health contract and PCM16 output as the old melo worker
- docker/setup-xtts.sh: builds the venv with cu128 torch (Blackwell) + Coqui
  TTS and bakes the XTTS-v2 model offline. Pins transformers>=4.57,<5 (5.x
  removed isin_mps_friendly, breaking XTTS) and installs the [codec] extra
  (torch>=2.9 needs torchcodec) — both verified by a real host synth
- Dockerfile: replace the melo build layer with the xtts layer
- supervisord.conf: melo-worker -> xtts-worker, env passthrough for
  XTTS_DEVICE/SPEAKER/LANGUAGE (always set via compose defaults)
- bridge/server.py: default TTS_ENGINE=xtts, route to the xtts worker, generic
  worker-synth helper, neural-only fallback flag (XTTS_FALLBACK_PIPER)
- settings UI: engine dropdown xtts/piper, drop the dead melo_speed field, fix
  the supervisorctl restart target to xtts-worker
- compose/.env.example/README: XTTS_* vars, speaker/language knobs, remove melo
- remove bridge/melo_worker.py and docker/setup-melo.sh
- tests: xtts treated as multilingual (not English-only)

Verified on host: coqui-tts loads XTTS-v2 and synthesises Korean as
"Ana Florence" to a 16-bit mono 24kHz WAV.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 03:08:01 +09:00
javis-bot
b9f637faa4 fix: stop hardcoding MELO_SPEED so the .env override reaches the worker
Some checks failed
Release / semantic-release (push) Successful in 26s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m13s
Release / build-linux (push) Failing after 7m8s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
supervisord.conf passed MELO_DEVICE through as %(ENV_MELO_DEVICE)s but pinned
MELO_SPEED="1.5", so lowering MELO_SPEED in .env had no effect — the worker
always got 1.5. Pass MELO_SPEED through with %(ENV_MELO_SPEED)s and set a
compose default (MELO_SPEED=${MELO_SPEED:-1.5}, same pattern as MELO_DEVICE) so
the supervisord expansion always resolves and an .env value actually changes
the speaking rate. Default rate is unchanged (1.5). melo_worker logs the
resolved speed at startup, so the env->worker path is verifiable.

Verified: _resolve_speed() returns 1.1 for MELO_SPEED=1.1 (1.5 otherwise), and
`MELO_SPEED=1.1 docker compose config` renders MELO_SPEED: "1.1" into the env.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 01:02:43 +09:00
javis-bot
2f000ac6c8 feat: load operator instructions from agents/*.md into the reply prompt
Some checks failed
Release / semantic-release (push) Successful in 30s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled
Drop Markdown files into an agents/ folder and their contents are appended to
the main reply LLM's system prompt, so an operator can extend the assistant's
rules/tone without code changes. Files are concatenated in filename order
(use 00-, 10- prefixes to control ordering) and re-read once per turn, so edits
apply on the next reply with no rebuild/restart. Fail-open: a missing, empty,
or unreadable folder yields no instructions and never breaks a reply.

- load_agent_instructions() in system_prompt.py (AGENTS_DIR env, default
  /app/agents); reads *.md only, skips blanks, ignores non-dir paths
- engine.py appends it alongside the existing settings-UI llm_instructions,
  under the same "Additional instructions from the operator:" framing
- docker-compose.yml bind-mounts ./agents:/app/agents:ro and sets AGENTS_DIR
- agents/example.md.sample starter template (.sample is not loaded)
- tests cover ordering, md-only filtering, blank-skip, env/arg resolution,
  and fail-open paths
- README, .env.example, docs/llm_contexts.md updated

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 00:57:54 +09:00
javis-bot
677bfcd2a9 feat: log the resolved whisper device on bridge load
Some checks failed
Release / semantic-release (push) Successful in 24s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m19s
Release / build-linux (push) Failing after 7m10s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
The bridge only logged Whisper's device on the CPU-fallback path, so a
successful GPU (or silent CPU) load was invisible. Print the CTranslate2-
resolved device on success and on the fallback load, so it is verifiable that
STT is actually running on cuda alongside ollama (GPU) and MeloTTS (cuda).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 00:19:20 +09:00
javis-bot
e49be6d04e fix: add video driver capability so NVENC works in the container
Some checks failed
Release / semantic-release (push) Successful in 31s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled
The Go-Live broadcast encodes with h264_nvenc, but the image only requested
NVIDIA_DRIVER_CAPABILITIES=compute,utility. The NVIDIA Container Toolkit gates
which driver libraries it injects by capability, and the NVENC/NVDEC libs
(libnvidia-encode.so.1 / libnvidia-decode.so.1) come with the `video`
capability. Without it the broadcast ffmpeg dies with
"Cannot load libnvidia-encode.so.1", the capture produces no packets, and
Go-Live never connects, while CUDA workloads (ollama/whisper/melo) and
nvidia-smi keep working because compute+utility are present.

Add `video` so hardware encode is available. Applies to both Linux (CDI) and
Windows Docker Desktop (WSL2).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 00:16:12 +09:00
javis-bot
1efabe03b1 fix: strip CR from container shell scripts in Dockerfile build
Some checks failed
Release / semantic-release (push) Successful in 26s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m12s
Release / build-linux (push) Failing after 7m8s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
.gitattributes pins *.sh to LF, but that only helps after a full working-tree
renormalise, which a Windows build box may not have done. The image build kept
failing at `RUN bash setup-melo.sh` because the checked-out file still had CRLF,
so bash read line 18 as `set -euxo pipefail\r` and aborted with
"set: pipefail: invalid option name".

Strip CR from setup-melo.sh before running it, and normalise all docker/scripts
shell scripts to LF after the app COPY so their shebangs (entrypoint, run-*.sh)
also survive a CRLF checkout. Makes the build EOL-agnostic regardless of host
autocrlf settings.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-22 23:35:12 +09:00
javis-bot
09cd4c5e31 fix: pin docker shell scripts to LF to stop CRLF breaking the image build
Some checks failed
Release / semantic-release (push) Successful in 24s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m18s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Windows checkouts with autocrlf=true inject CR into docker/*.sh, so inside
the Linux container `set -euxo pipefail` is read as `pipefail\r` and bash
aborts with "set: pipefail: invalid option name", failing setup-melo.sh and
the whole image build. .gitattributes already pinned .bat/.cmd/.ps1 to CRLF
but never pinned .sh, leaving all nine container scripts exposed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-22 23:27:46 +09:00
javis-bot
00ce813845 docs: warn that COMPOSE_FILE uses ';' on Windows, ':' on Linux/macOS
Some checks failed
Release / semantic-release (push) Successful in 26s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m21s
Release / build-linux (push) Failing after 7m6s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Windows users following the docs hit "The system cannot find the file
specified" because COMPOSE_FILE's separator is OS-specific (':' collides
with the C: drive letter). Fix every Windows example to use ';', add an
explicit OS-separator warning in .env.example, README, DEPLOY.md and the
gpu-windows compose comment, and point users at the explicit `-f` form as
a separator-agnostic alternative.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-22 23:13:33 +09:00
javis-bot
c56ce1eb30 feat: human-like typing for browser Google and YouTube search
Some checks failed
Release / semantic-release (push) Successful in 26s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m16s
Release / build-linux (push) Failing after 7m11s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Make the browser search helper search the way a person does: load the
site home page, type the query into the search box one key at a time, and
press Enter — for both Google `search` and `youtube` — instead of jumping
straight to a results URL. Supports the goal of a human-like assistant.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-22 21:09:15 +09:00
javis-bot
597207dd33 feat: reuse a signed-in Chrome profile for browser web search
Add CHROME_USER_DATA_DIR so the browser search fallback can open Chrome
against a dedicated, Google-signed-in profile instead of a fresh anonymous
session. A returning signed-in profile is what actually avoids Google's
/sorry bot-detection page, so this is the reliable way to get browser
Google search in plain text turns. Fallback order is now CDP (broadcast
Chrome) -> persistent profile (when configured) -> ephemeral headless,
all still fail-open to the DDG/Brave/Wikipedia cascade.

Document the profile in .env.example and web_search.spec.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-22 20:57:25 +09:00
javis-bot
98a1825d01 feat: headless Chrome fallback for browser web search outside broadcast
browse-search.mjs only connected to the on-screen broadcast Chrome over
CDP, so browser-based Google search worked only during a live broadcast;
plain text turns fell through to the DDG cascade. Add a headless fallback
(system Chrome via channel:'chrome', else Playwright's bundled chromium)
for `search` mode so general conversation can use Google at no API cost.
`youtube` still requires the visible broadcast Chrome.

Detect Google's /sorry bot-detection interstitial structurally by URL and
fail fast so the caller fail-opens to DDG/Brave/Wikipedia instead of
treating the challenge page as empty results.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-22 20:52:54 +09:00
javis-bot
da27c5a306 docs: warn that personal Google login is blocked on the Gemini CLI path
Google now rejects personal Google accounts on the Gemini CLI OAuth login
("This client is no longer supported for Gemini Code Assist for individuals").
The setup docs previously sent every user down "Sign in with Google" with no
warning. Note the block, recommend GEMINI_AUTH=apikey for personal accounts,
and clarify that real-time search fail-opens to DDG/Brave/Wikipedia regardless.

Docs only; no runtime default change.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-22 19:44:12 +09:00
22 changed files with 529 additions and 115 deletions

View File

@@ -34,18 +34,18 @@ WHISPER_DEVICE=cuda
WHISPER_COMPUTE_TYPE=float16 WHISPER_COMPUTE_TYPE=float16
# Optional explicit Piper voice model (.onnx). If empty, the jarvis default is used. # Optional explicit Piper voice model (.onnx). If empty, the jarvis default is used.
TTS_PIPER_MODEL_PATH= TTS_PIPER_MODEL_PATH=
# TTS engine: "melo" (default) uses the MeloTTS Korean voice served by the warm # TTS engine: "edge" (default) uses Microsoft Edge TTS, a natural Korean neural
# melo-worker (Korean speaker, speed 1.5). Set to "piper" to use Piper directly. # voice. Set to "piper" for the offline English voice. NOTE: edge is ONLINE —
TTS_ENGINE=melo # reply text is sent to Microsoft's servers and needs internet.
# Melo-only by default: if MeloTTS synthesis fails the bridge returns no audio TTS_ENGINE=edge
# rather than speaking Korean through the English Piper voice (which mangles it). # Edge voice + speaking rate. Rate is a percentage (+45% ≈ 1.45×). Korean voices:
# Set to 1 only if you explicitly want the Piper fallback. # ko-KR-HyunsuMultilingualNeural (M), ko-KR-InJoonNeural (M), ko-KR-SunHiNeural (F).
EDGE_TTS_VOICE=ko-KR-HyunsuMultilingualNeural
EDGE_TTS_RATE=+45%
# Neural-only by default: if synthesis fails the bridge returns no audio rather
# than speaking Korean through the English Piper voice. Set to 1 to allow the
# Piper fallback.
MELO_FALLBACK_PIPER=0 MELO_FALLBACK_PIPER=0
# Where the bridge reaches the in-container MeloTTS worker, and how long it
# waits for a synthesis. Speaking rate is set on the worker via MELO_SPEED.
MELO_WORKER_URL=http://127.0.0.1:8770
MELO_TIMEOUT=30
MELO_SPEED=1.5
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Jarvis brain (Ollama-backed). In Docker these populate the rendered # Jarvis brain (Ollama-backed). In Docker these populate the rendered
@@ -74,6 +74,12 @@ WHISPER_MODEL=small
# occasional trailing CJK fragment small models leak on free-form chat). # occasional trailing CJK fragment small models leak on free-form chat).
OUTPUT_LANGUAGE= OUTPUT_LANGUAGE=
# Operator instruction folder: every *.md in this dir is appended to the main
# reply LLM's system prompt (filename order), re-read each turn so edits apply
# without a rebuild/restart. ./agents is bind-mounted here read-only; only
# change this to relocate the folder inside the container. See README "운영자 지시문".
AGENTS_DIR=/app/agents
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Docker desktop (VNC) — used only by the container image # Docker desktop (VNC) — used only by the container image
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -98,12 +104,28 @@ CHROME_START_URL=about:blank
# on-screen browser for real-time info (search / play / read screen). # on-screen browser for real-time info (search / play / read screen).
# false = no screen share; voice only, real-time info via the Gemini API. # false = no screen share; voice only, real-time info via the Gemini API.
STREAM_BROWSER=true STREAM_BROWSER=true
# Optional: profile dir for browser-based Google search in plain text turns
# (no active broadcast). When set, the search helper opens Chrome against this
# profile instead of a fresh anonymous one. Sign that profile into Google once
# (run a real Chrome with --user-data-dir=<this path> and log in) so Google
# treats later searches as a returning user and does not serve the bot-detection
# page. Leave blank to use an ephemeral headless session (works only where
# Google does not challenge it). Use a DEDICATED dir, not your everyday Chrome
# profile, to avoid the "profile in use" lock while Chrome is open.
CHROME_USER_DATA_DIR=
# Gemini auth for real-time info when STREAM_BROWSER=false. # Gemini auth for real-time info when STREAM_BROWSER=false.
# oauth = use the Gemini CLI with a Google-account login (no API key). # oauth = use the Gemini CLI with a Google-account login (no API key).
# Install once: npm i -g @google/gemini-cli ; then run `gemini` and # Install once: npm i -g @google/gemini-cli ; then run `gemini` and
# "Sign in with Google". Uses the CLI's built-in web-search grounding. # "Sign in with Google". Uses the CLI's built-in web-search grounding.
# apikey = legacy REST path; needs GEMINI_API_KEY below # NOTE (2026-06): Google is blocking personal Google accounts on this
# (get one at https://aistudio.google.com/app/apikey). # path ("This client is no longer supported for Gemini Code Assist for
# individuals"). Workspace/org accounts may still work; personal
# accounts should use apikey below instead.
# apikey = REST path; needs GEMINI_API_KEY below
# (get one at https://aistudio.google.com/app/apikey). Recommended for
# personal Google accounts now that individual OAuth login is blocked.
# Either way, real-time search fail-opens to DDG/Brave/Wikipedia if Gemini is
# unavailable, so this is optional, not required.
GEMINI_AUTH=oauth GEMINI_AUTH=oauth
GEMINI_API_KEY= GEMINI_API_KEY=
GEMINI_MODEL=gemini-2.0-flash GEMINI_MODEL=gemini-2.0-flash
@@ -174,11 +196,18 @@ VOICE_SILENCE_MS=800
JARVIS_ROLE=full JARVIS_ROLE=full
# --- GPU per OS: pick the matching compose override via COMPOSE_FILE --- # --- GPU per OS: pick the matching compose override via COMPOSE_FILE ---
# Ubuntu (nvidia-container-toolkit / CDI): # IMPORTANT: the file separator is OS-specific. Linux/macOS use ":" (colon);
# Windows uses ";" (semicolon), because ":" is taken by the drive letter (C:).
# Using the wrong one makes Docker treat the whole string as a single missing
# filename ("...gpu-windows.yml: The system cannot find the file specified").
# Ubuntu / macOS (nvidia-container-toolkit / CDI):
# COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml # COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml
# Windows 11 (Docker Desktop + WSL2 + NVIDIA): # Windows 11 (Docker Desktop + WSL2 + NVIDIA) — note the ";" separator:
# COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-windows.yml # COMPOSE_FILE=docker-compose.yml;docker-compose.gpu-windows.yml
# Browser-only host (no GPU needed): leave COMPOSE_FILE unset (base only). # Browser-only host (no GPU needed): leave COMPOSE_FILE unset (base only).
# Default below is the Linux form; Windows users must change ":" to ";" AND
# swap gpu-linux for gpu-windows. If unsure, comment this out and pass the
# files explicitly: docker compose -f docker-compose.yml -f <gpu-override> ...
COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml
# --- Browser HOST (JARVIS_ROLE=browser) — e.g. this LAN machine --- # --- Browser HOST (JARVIS_ROLE=browser) — e.g. this LAN machine ---

6
.gitattributes vendored
View File

@@ -7,3 +7,9 @@
# PowerShell is more forgiving but the same logic applies. # PowerShell is more forgiving but the same logic applies.
*.ps1 text eol=crlf *.ps1 text eol=crlf
# Shell scripts run inside the Linux container; they MUST stay LF even when
# checked out on Windows. autocrlf=true would otherwise inject CR and break
# `set -o pipefail`, shebangs, and heredocs (e.g. docker/setup-melo.sh failing
# the image build with "set: pipefail: invalid option name").
*.sh text eol=lf

View File

@@ -10,8 +10,14 @@ ENV DEBIAN_FRONTEND=noninteractive \
DISPLAY=:1 \ DISPLAY=:1 \
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 \ PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 \
PATH=/opt/venv/bin:/root/.bun/bin:/usr/local/bin:/usr/bin:/bin \ PATH=/opt/venv/bin:/root/.bun/bin:/usr/local/bin:/usr/bin:/bin \
NVIDIA_VISIBLE_DEVICES=all \ NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=compute,utility
# `video` is REQUIRED for NVENC/NVDEC: it tells the NVIDIA Container Toolkit to
# inject libnvidia-encode.so.1 / libnvidia-decode.so.1 into the container. With
# only `compute,utility` you get CUDA (ollama/whisper/melo) + nvidia-smi, but the
# Go-Live broadcast's h264_nvenc fails with "Cannot load libnvidia-encode.so.1".
# Applies on both Linux (CDI) and Windows Docker Desktop (WSL2).
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
# --- System packages: desktop, VNC, Chrome deps, ffmpeg, python, ocr --- # --- System packages: desktop, VNC, Chrome deps, ffmpeg, python, ocr ---
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -59,16 +65,14 @@ RUN ls -d /opt/venv/lib/python*/site-packages/nvidia/cublas/lib \
> /etc/ld.so.conf.d/nvidia-cu12.conf 2>/dev/null \ > /etc/ld.so.conf.d/nvidia-cu12.conf 2>/dev/null \
&& /sbin/ldconfig || true && /sbin/ldconfig || true
# --- MeloTTS Korean voice (separate /opt/melo py3.11 venv; see setup-melo.sh). # --- Korean voice: Microsoft Edge TTS (online neural). No model is baked — the
# Heavy layer (torch CPU + transformers + MeCab); placed before the app # `edge-tts` pip package (in requirements-bridge.txt) calls the MS service at
# COPY so it stays cached across source-only changes. --- # runtime and the bridge transcodes the MP3 to PCM16 with ffmpeg. No heavy
COPY docker/setup-melo.sh /app/docker/setup-melo.sh # TTS build layer is needed. ---
RUN bash /app/docker/setup-melo.sh
# --- Human input + window management for the on-screen Chrome control tool. # --- Human input + window management for the on-screen Chrome control tool.
# Placed AFTER the heavy melo layer so it doesn't bust that cache. xdotool # xdotool injects real X pointer/keyboard events (visible cursor,
# injects real X pointer/keyboard events (visible cursor, char-by-char # char-by-char typing) into the broadcast; wmctrl lists/moves windows. ---
# typing) into the broadcast; wmctrl lists/moves windows. ---
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
xdotool wmctrl \ xdotool wmctrl \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
@@ -81,6 +85,11 @@ RUN cd /app/bot && bun install --frozen-lockfile || bun install
COPY . /app COPY . /app
WORKDIR /app WORKDIR /app
# Normalise all container shell scripts to LF. On a Windows checkout (autocrlf)
# these arrive as CRLF, which would break their shebangs at runtime (entrypoint,
# run-*.sh) the same way it broke setup-melo.sh at build time.
RUN find /app/docker /app/scripts -name '*.sh' -exec sed -i 's/\r$//' {} +
# --- Default Piper voice (best-effort at build; entrypoint retries if absent) --- # --- Default Piper voice (best-effort at build; entrypoint retries if absent) ---
RUN bash docker/download-piper.sh || true RUN bash docker/download-piper.sh || true

View File

@@ -69,19 +69,21 @@ docker compose -f docker-compose.yml -f docker-compose.gpu-linux.yml up -d --bui
docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d --build docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d --build
# ── GPU 없이 (CPU 전용 호스트) ── # ── GPU 없이 (CPU 전용 호스트) ──
# .env 에 WHISPER_DEVICE=cpu, MELO_DEVICE=cpu 를 넣고 베이스만 사용 # .env 에 WHISPER_DEVICE=cpu 를 넣고 베이스만 사용
docker compose up -d --build docker compose up -d --build
``` ```
매번 `-f`를 치기 싫으면 `.env`에 한 줄 넣어두면 그냥 `docker compose up -d`로 됩니다(override가 자동 적용): 매번 `-f`를 치기 싫으면 `.env`에 한 줄 넣어두면 그냥 `docker compose up -d`로 됩니다(override가 자동 적용):
```bash ```bash
# Linux # Linux / macOS (구분자 = 콜론 ":")
COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml
# Windows 11 # Windows 11 (구분자 = 세미콜론 ";" — 콜론은 드라이브 문자 C: 와 충돌)
COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-windows.yml COMPOSE_FILE=docker-compose.yml;docker-compose.gpu-windows.yml
``` ```
> ⚠️ `COMPOSE_FILE`의 파일 구분자는 OS마다 다릅니다: Linux/macOS는 `:`, Windows는 `;`. Windows에서 `:`를 쓰면 Docker가 전체를 파일 하나 이름으로 읽어 `... The system cannot find the file specified` 에러가 납니다. 헷갈리면 `COMPOSE_FILE`을 비워두고 실행 시 직접 지정하세요: `docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d --build`.
> Linux와 Windows는 GPU를 컨테이너에 넣는 방식이 달라서 override 파일이 갈립니다. Linux는 CDI(`devices: nvidia.com/gpu=all`), Windows(Docker Desktop)는 Compose의 `deploy.resources.reservations.devices`(`driver: nvidia`)를 씁니다. 호스트 사전 준비는 아래 "GPU 가속" 절 참고. > Linux와 Windows는 GPU를 컨테이너에 넣는 방식이 달라서 override 파일이 갈립니다. Linux는 CDI(`devices: nvidia.com/gpu=all`), Windows(Docker Desktop)는 Compose의 `deploy.resources.reservations.devices`(`driver: nvidia`)를 씁니다. 호스트 사전 준비는 아래 "GPU 가속" 절 참고.
`docker compose up` 한 번이면 자동으로: `docker compose up` 한 번이면 자동으로:
@@ -111,7 +113,7 @@ docker compose up -d # 유저봇이 로그인해 지정 음성채널에
### GPU 가속 (OS별) ### GPU 가속 (OS별)
LLM(Ollama), Whisper STT, MeloTTS가 GPU에서 돕니다(env 기본 `WHISPER_DEVICE=cuda`, `MELO_DEVICE=cuda`). NVIDIA Blackwell(sm_120, 예: RTX 5050/5070Ti)에서 검증: 컨테이너 내 torch cu128 CUDA 동작, Ollama GPU 오프로드, faster-whisper float16, MeloTTS GPU 합성 모두 확인. LLM(Ollama) Whisper STT가 GPU에서 돕니다(env 기본 `WHISPER_DEVICE=cuda`). TTS는 기본값이 Edge TTS(온라인 한국어 음성)라 GPU를 쓰지 않습니다. NVIDIA Blackwell(sm_120, 예: RTX 5050/5070Ti)에서 검증: 컨테이너 내 torch cu128 CUDA 동작, Ollama GPU 오프로드, faster-whisper float16 모두 확인.
GPU는 위 "실행 — Docker"의 OS별 override 파일로 켜집니다. 호스트 사전 준비는 OS마다 다릅니다: GPU는 위 "실행 — Docker"의 OS별 override 파일로 켜집니다. 호스트 사전 준비는 OS마다 다릅니다:
@@ -135,7 +137,7 @@ docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L # GPU 보이
**공통:** **공통:**
- 모델: 베이스 compose 기본은 `qwen2.5:3b`(8GB VRAM에서 도구호출 안정적). 더 무겁게(`qwen2.5:7b`, `qwen3:8b` 등) 쓰려면 `.env``OLLAMA_CHAT_MODEL` 변경. - 모델: 베이스 compose 기본은 `qwen2.5:3b`(8GB VRAM에서 도구호출 안정적). 더 무겁게(`qwen2.5:7b`, `qwen3:8b` 등) 쓰려면 `.env``OLLAMA_CHAT_MODEL` 변경.
- **GPU가 없거나 인식 실패 시 자동으로 CPU 폴백**(Whisper)하므로 안전합니다. 명시적으로 CPU만 쓰려면 override 파일 없이 베이스만 올리고 `.env``WHISPER_DEVICE=cpu`, `MELO_DEVICE=cpu`를 두세요. - **GPU가 없거나 인식 실패 시 자동으로 CPU 폴백**(Whisper)하므로 안전합니다. 명시적으로 CPU만 쓰려면 override 파일 없이 베이스만 올리고 `.env``WHISPER_DEVICE=cpu`를 두세요.
- 데이터(메모리 DB), Whisper 캐시, Piper 음성은 named volume에 영속됩니다. - 데이터(메모리 DB), Whisper 캐시, Piper 음성은 named volume에 영속됩니다.
- 셀프봇 영상 송출 의존성은 이미지에 기본 포함하지 않습니다. 쓰려면 컨테이너에서 `cd /app/bot && bun add discord.js-selfbot-v13 @dank074/discord-video-stream` 후 재시작(또는 Dockerfile에 추가). - 셀프봇 영상 송출 의존성은 이미지에 기본 포함하지 않습니다. 쓰려면 컨테이너에서 `cd /app/bot && bun add discord.js-selfbot-v13 @dank074/discord-video-stream` 후 재시작(또는 Dockerfile에 추가).
@@ -241,10 +243,22 @@ cd bot; bun run register; bun run start # 창 2: (일반 봇이면) 슬래시
- `BRIDGE_URL` — 봇이 호출할 브릿지 주소 (기본 `http://127.0.0.1:8765`) - `BRIDGE_URL` — 봇이 호출할 브릿지 주소 (기본 `http://127.0.0.1:8765`)
- `STREAM_BACKEND`, `DISCORD_SELFBOT_TOKEN`, `NOVNC_URL` — 화면 송출 - `STREAM_BACKEND`, `DISCORD_SELFBOT_TOKEN`, `NOVNC_URL` — 화면 송출
- `VNC_DISPLAY=:1`, `VNC_RESOLUTION`, `VNC_FRAMERATE`, `VNC_BITRATE_KBPS` — 캡처 - `VNC_DISPLAY=:1`, `VNC_RESOLUTION`, `VNC_FRAMERATE`, `VNC_BITRATE_KBPS` — 캡처
- `WHISPER_DEVICE/COMPUTE_TYPE`, `MELO_DEVICE` — GPU 호스트면 `cuda`/`float16`, CPU 전용이면 `cpu`(GPU 자체는 OS별 override compose 파일로 켬) - `WHISPER_DEVICE/COMPUTE_TYPE` — GPU 호스트면 `cuda`/`float16`, CPU 전용이면 `cpu`(GPU 자체는 OS별 override compose 파일로 켬)
- `OLLAMA_CHAT_MODEL` — 두뇌 LLM (기본 `qwen2.5:3b`) - `OLLAMA_CHAT_MODEL` — 두뇌 LLM (기본 `qwen2.5:3b`)
- `COMPOSE_FILE` — OS별 GPU override를 매번 `-f`로 안 치고 자동 적용 (위 "실행 — Docker" 참고) - `COMPOSE_FILE` — OS별 GPU override를 매번 `-f`로 안 치고 자동 적용 (위 "실행 — Docker" 참고)
- `output_language` — 출력 언어 고정(비우면 사용자 언어). 설정 웹 UI(`/settings`)에서 바꾸면 env 기본값보다 우선하며 컨테이너 재생성 후에도 유지됩니다. - `output_language` — 출력 언어 고정(비우면 사용자 언어). 설정 웹 UI(`/settings`)에서 바꾸면 env 기본값보다 우선하며 컨테이너 재생성 후에도 유지됩니다.
- `AGENTS_DIR` — 운영자 지시문 폴더(기본 `/app/agents`, `./agents`가 read-only로 마운트됨). 아래 "운영자 지시문" 참고.
---
## 운영자 지시문 (`agents/*.md`)
`agents/` 폴더에 마크다운 파일을 넣으면 그 내용이 어시스턴트의 메인 답변 시스템 프롬프트 뒤에 그대로 추가됩니다. 페르소나(집사 성격)는 그대로 두고 규칙·말투·금칙어 등을 덧붙일 때 쓰세요.
- `agents/` 안의 모든 `*.md`를 **파일명 순서**로 이어 붙입니다. 순서를 정하려면 `00-tone.md`, `10-rules.md`처럼 숫자 접두사를 쓰세요.
- **매 답변마다 다시 읽습니다.** 파일을 저장하면 다음 발화부터 바로 반영되며, 재빌드/재시작이 필요 없습니다(폴더가 read-only로 마운트됨).
- 폴더가 없거나 비어 있으면 아무 일도 일어나지 않습니다(fail-open).
- `agents/example.md.sample`을 `rules.md` 등 `*.md`로 복사해서 시작하세요. `.sample` 파일은 로드되지 않습니다.
--- ---

15
agents/example.md.sample Normal file
View File

@@ -0,0 +1,15 @@
# Operator instruction file (example)
#
# HOW TO USE: copy or rename this file to anything ending in `.md`
# (e.g. `rules.md`). Every `*.md` in this folder is appended to the assistant's
# main reply system prompt, in filename order — use number prefixes like
# `00-tone.md`, `10-rules.md` to control ordering. Edits take effect on the
# NEXT reply; no rebuild or restart is needed (the folder is read per turn).
#
# Files ending in `.sample` (like this one) are ignored, so this template never
# affects replies until you rename it to `*.md`.
#
# Everything below a heading is treated as plain instruction text for the LLM.
Always keep replies under two sentences.
When the user asks about deployment, mention the relevant docker compose command.

View File

@@ -1,33 +1,112 @@
// True-mode browser action core. Drives the on-screen Chrome (CDP at CDP_PORT, // Browser action core. Prefers the on-screen Chrome (CDP at CDP_PORT, default
// default 9222) so the action is visible on the Go-Live broadcast, and prints a // 9222) so the action is visible on the Go-Live broadcast, and prints a JSON
// JSON result on stdout for the Python `browseAndSearch` tool to wrap. // result on stdout for the Python `browseAndSearch` tool to wrap.
// //
// node browse-search.mjs "<query>" [search|youtube] // node browse-search.mjs "<query>" [search|youtube]
// //
// - search : Google-search the query, return the top organic results. // - search : Google-search the query, return the top organic results.
// - youtube : search YouTube and play the first result. // - youtube : search YouTube and play the first result.
//
// Backend selection for `search`:
// 1. The broadcast Chrome over CDP (visible on the Go-Live stream).
// 2. Else, if CHROME_USER_DATA_DIR is set, a persistent Chrome using that
// profile dir. Logging that dedicated profile into Google once lets Google
// treat later searches as a returning signed-in user, which avoids the
// bot-detection interstitial that blocks a fresh anonymous session.
// 3. Else a fresh ephemeral headless Chrome (works only where Google does not
// challenge the session, e.g. a non-flagged residential IP).
// `youtube` only makes sense on the visible broadcast Chrome, so it never uses
// the headless/persistent fallback.
import { chromium } from 'playwright'; import { chromium } from 'playwright';
const CDP = process.env.CDP_PORT || '9222'; const CDP = process.env.CDP_PORT || '9222';
// Use 127.0.0.1, not "localhost": in containers localhost can resolve to IPv6 // Use 127.0.0.1, not "localhost": in containers localhost can resolve to IPv6
// (::1) first while Chrome's CDP listens on IPv4, giving ECONNREFUSED ::1. // (::1) first while Chrome's CDP listens on IPv4, giving ECONNREFUSED ::1.
const CDP_HOST = process.env.CDP_HOST || '127.0.0.1'; const CDP_HOST = process.env.CDP_HOST || '127.0.0.1';
const USER_DATA_DIR = process.env.CHROME_USER_DATA_DIR || '';
const UA =
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
'(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36';
const query = process.argv[2] || ''; const query = process.argv[2] || '';
const mode = (process.argv[3] || 'search').toLowerCase(); const mode = (process.argv[3] || 'search').toLowerCase();
const out = (o) => { process.stdout.write(JSON.stringify(o)); }; const out = (o) => { process.stdout.write(JSON.stringify(o)); };
if (!query) { out({ ok: false, error: 'no query' }); process.exit(1); } if (!query) { out({ ok: false, error: 'no query' }); process.exit(1); }
let b; let connected; // CDP Browser (the broadcast Chrome — never kill it)
let launchedBrowser; // ephemeral headless Browser we launched
let persistent; // persistent BrowserContext we launched
let launched = false;
let page;
// Try system Chrome (channel:'chrome') first so no extra Playwright browser
// download is needed; fall back to Playwright's bundled chromium.
async function tryLaunch(launchFn) {
let err;
for (const opts of [{ headless: true, channel: 'chrome' }, { headless: true }]) {
try { try {
b = await chromium.connectOverCDP(`http://${CDP_HOST}:${CDP}`); return await launchFn(opts);
const ctx = b.contexts()[0]; } catch (e) {
const page = ctx.pages()[0] || (await ctx.newPage()); err = e;
}
}
throw err;
}
async function acquirePage() {
// 1. Broadcast Chrome over CDP.
try {
connected = await chromium.connectOverCDP(`http://${CDP_HOST}:${CDP}`);
const ctx = connected.contexts()[0];
page = ctx.pages()[0] || (await ctx.newPage());
return;
} catch (e) {
if (mode === 'youtube') throw e; // youtube needs the visible broadcast Chrome
}
// 2. Persistent profile (signed-in) when configured.
if (USER_DATA_DIR) {
persistent = await tryLaunch((opts) =>
chromium.launchPersistentContext(USER_DATA_DIR, { ...opts, locale: 'ko-KR', userAgent: UA }),
);
launched = true;
page = persistent.pages()[0] || (await persistent.newPage());
return;
}
// 3. Ephemeral headless.
launchedBrowser = await tryLaunch((opts) => chromium.launch(opts));
launched = true;
const ctx = await launchedBrowser.newContext({ locale: 'ko-KR', userAgent: UA });
page = await ctx.newPage();
}
async function closeAll() {
try { await persistent?.close(); } catch { /* ignore */ }
try { await launchedBrowser?.close(); } catch { /* ignore */ }
try { await connected?.close(); } catch { /* ignore */ }
}
// Human-like search: land on the site's home page, type the query into its
// search box one key at a time, and press Enter — the way a person would,
// rather than jumping straight to a results URL.
async function typeSearch(homeUrl, boxSelector, query) {
await page.goto(homeUrl, { waitUntil: 'domcontentloaded' });
const box = page.locator(boxSelector).first();
await box.waitFor({ timeout: 15000 });
await box.click();
await box.pressSequentially(query, { delay: 45 });
await box.press('Enter');
}
try {
await acquirePage();
page.setDefaultTimeout(20000); page.setDefaultTimeout(20000);
await page.bringToFront().catch(() => {}); await page.bringToFront().catch(() => {});
if (mode === 'youtube') { if (mode === 'youtube') {
await page.goto(`https://www.youtube.com/results?search_query=${encodeURIComponent(query)}`, { waitUntil: 'domcontentloaded' }); // Type into YouTube's search box like a person, then play the first result.
await typeSearch('https://www.youtube.com/?hl=ko', 'input#search, input[name="search_query"]', query);
await page.waitForSelector('ytd-video-renderer a#video-title, a#video-title', { timeout: 20000 }); await page.waitForSelector('ytd-video-renderer a#video-title, a#video-title', { timeout: 20000 });
const first = page.locator('ytd-video-renderer a#video-title, a#video-title').first(); const first = page.locator('ytd-video-renderer a#video-title, a#video-title').first();
const title = (await first.getAttribute('title').catch(() => '')) || (await first.innerText().catch(() => '')); const title = (await first.getAttribute('title').catch(() => '')) || (await first.innerText().catch(() => ''));
@@ -36,8 +115,19 @@ try {
await page.evaluate(() => { const v = document.querySelector('video'); if (v && v.paused) v.play(); }); await page.evaluate(() => { const v = document.querySelector('video'); if (v && v.paused) v.play(); });
out({ ok: true, mode, title: (title || '').trim(), url: page.url() }); out({ ok: true, mode, title: (title || '').trim(), url: page.url() });
} else { } else {
await page.goto(`https://www.google.com/search?q=${encodeURIComponent(query)}&hl=ko`, { waitUntil: 'domcontentloaded' }); // Type into Google's search box like a person, then read the results.
await typeSearch('https://www.google.com/?hl=ko', 'textarea[name="q"], input[name="q"]', query);
await page.waitForLoadState('domcontentloaded');
await page.waitForTimeout(1500); await page.waitForTimeout(1500);
// Google serves its bot-detection interstitial (/sorry/index) to sessions it
// suspects are automated. Detect it structurally (by URL, locale-independent)
// and fail fast so the Python caller fail-opens to the DDG cascade instead of
// treating an empty challenge page as "no results".
if (page.url().includes('/sorry/')) {
await closeAll();
out({ ok: false, error: 'google-bot-challenge', headless: launched });
process.exit(1);
}
const results = await page.evaluate(() => { const results = await page.evaluate(() => {
const seen = new Set(); const seen = new Set();
const items = []; const items = [];
@@ -55,11 +145,11 @@ try {
} }
return items; return items;
}); });
out({ ok: true, mode, query, count: results.length, results }); out({ ok: true, mode, query, count: results.length, results, headless: launched });
} }
await b.close(); await closeAll();
} catch (e) { } catch (e) {
try { await b?.close(); } catch { /* ignore */ } await closeAll();
out({ ok: false, error: String(e?.message || e) }); out({ ok: false, error: String(e?.message || e) });
process.exit(1); process.exit(1);
} }

View File

@@ -21,7 +21,11 @@ nvidia-cudnn-cu12
# --- Bridge HTTP service --- # --- Bridge HTTP service ---
flask>=3.0.0 flask>=3.0.0
# --- Text-to-speech (Piper) --- # --- Text-to-speech ---
# Edge TTS: the primary Korean voice (online MS neural). Lightweight (httpx);
# emits MP3, transcoded to PCM16 by the system ffmpeg in the bridge.
edge-tts>=6.1.0
# Piper: offline English fallback.
piper-tts>=1.3.0 piper-tts>=1.3.0
# --- Built-in tools (lazily imported; needed for full functionality) --- # --- Built-in tools (lazily imported; needed for full functionality) ---

View File

@@ -87,12 +87,11 @@ VAD_MIN_SPEECH_MS = int(os.environ.get("VAD_MIN_SPEECH_MS", "200"))
# Korean phrase decoded as Chinese) and shaves a little latency. Empty = auto. # Korean phrase decoded as Chinese) and shaves a little latency. Empty = auto.
STT_LANGUAGE = os.environ.get("STT_LANGUAGE", "ko").strip() or None STT_LANGUAGE = os.environ.get("STT_LANGUAGE", "ko").strip() or None
# TTS engine: "melo" (MeloTTS Korean speaker, the warm worker) is the primary # TTS engine: "edge" (Microsoft Edge TTS, natural Korean neural voice) is the
# voice; Piper is kept as a fallback if the worker is unreachable. Set # primary voice. "melo" (a warm MeloTTS worker) and "piper" remain selectable.
# TTS_ENGINE=piper to disable MeloTTS entirely.
def _tts_engine_setting() -> str: def _tts_engine_setting() -> str:
"""TTS engine: settings-UI value (runtime config JSON) wins, else env, else """TTS engine: settings-UI value (runtime config JSON) wins, else env, else
melo. Read at startup; the settings UI restarts the bridge on apply.""" edge. Read at startup; the settings UI restarts the bridge on apply."""
try: try:
_cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json") _cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json")
_v = json.loads(open(_cp, encoding="utf-8").read()).get("tts_engine") _v = json.loads(open(_cp, encoding="utf-8").read()).get("tts_engine")
@@ -100,16 +99,22 @@ def _tts_engine_setting() -> str:
return str(_v).strip().lower() return str(_v).strip().lower()
except Exception: except Exception:
pass pass
return os.environ.get("TTS_ENGINE", "melo").strip().lower() return os.environ.get("TTS_ENGINE", "edge").strip().lower()
TTS_ENGINE = _tts_engine_setting() TTS_ENGINE = _tts_engine_setting()
# Edge TTS (online MS neural voice). Voice + rate are env-driven so they can be
# changed without code. Default: Korean "Hyunsu" multilingual voice at +45%
# (≈1.45×), the chosen settings. NOTE: edge synthesis sends the reply TEXT to
# Microsoft's servers and needs internet — an intentional privacy trade-off for
# the more natural voice.
EDGE_TTS_VOICE = os.environ.get("EDGE_TTS_VOICE", "ko-KR-HyunsuMultilingualNeural").strip()
EDGE_TTS_RATE = os.environ.get("EDGE_TTS_RATE", "+45%").strip()
MELO_WORKER_URL = os.environ.get("MELO_WORKER_URL", "http://127.0.0.1:8770") MELO_WORKER_URL = os.environ.get("MELO_WORKER_URL", "http://127.0.0.1:8770")
MELO_TIMEOUT = float(os.environ.get("MELO_TIMEOUT", "30")) MELO_TIMEOUT = float(os.environ.get("MELO_TIMEOUT", "30"))
# When MeloTTS is the engine, do NOT silently fall back to the English Piper # Do NOT silently fall back to the English Piper voice on a neural-voice failure:
# voice on failure: speaking Korean text through an English voice produces # speaking Korean through an English voice produces mangled audio. Default is
# mangled audio. Default is melo-only (return no audio on failure); set # neural-only (return no audio on failure); set MELO_FALLBACK_PIPER=1 to opt in.
# MELO_FALLBACK_PIPER=1 to opt into the Piper fallback.
MELO_FALLBACK_PIPER = os.environ.get("MELO_FALLBACK_PIPER", "0") in ("1", "true", "True", "yes", "on") MELO_FALLBACK_PIPER = os.environ.get("MELO_FALLBACK_PIPER", "0") in ("1", "true", "True", "yes", "on")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -150,12 +155,17 @@ def _ensure_brain():
compute = os.environ.get("WHISPER_COMPUTE_TYPE", "auto") compute = os.environ.get("WHISPER_COMPUTE_TYPE", "auto")
try: try:
whisper = WhisperModel(cfg.whisper_model, device=device, compute_type=compute) whisper = WhisperModel(cfg.whisper_model, device=device, compute_type=compute)
# Log the device actually resolved by CTranslate2 (device="auto"
# picks cuda when available) so a silent CPU load is visible.
resolved = str(getattr(getattr(whisper, "model", None), "device", device)).lower()
print(f"[bridge] whisper loaded on {resolved} (compute={compute})", flush=True)
except Exception as ge: except Exception as ge:
# GPU not available / unsupported -> fall back to CPU so the # GPU not available / unsupported -> fall back to CPU so the
# bridge still works without a GPU passed to the container. # bridge still works without a GPU passed to the container.
if device != "cpu": if device != "cpu":
print(f"[bridge] whisper device='{device}' failed ({ge}); falling back to CPU", flush=True) print(f"[bridge] whisper device='{device}' failed ({ge}); falling back to CPU", flush=True)
whisper = WhisperModel(cfg.whisper_model, device="cpu", compute_type="int8") whisper = WhisperModel(cfg.whisper_model, device="cpu", compute_type="int8")
print("[bridge] whisper loaded on cpu (compute=int8)", flush=True)
else: else:
raise raise
@@ -297,6 +307,54 @@ def _coerce_bool(value) -> Optional[bool]:
return str(value).strip().lower() in ("1", "true", "yes", "on") return str(value).strip().lower() in ("1", "true", "yes", "on")
def _edge_synthesize(text: str) -> Optional[bytes]:
"""Synthesise via Microsoft Edge TTS (online neural voice) and return a
16-bit PCM WAV, or None on any failure. Edge emits MP3; we transcode to
PCM16 mono with the system ffmpeg, writing to a temp file (seekable) so the
WAV header carries a correct length. Needs internet."""
import asyncio
import subprocess
import tempfile
try:
import edge_tts # type: ignore
async def _gen() -> bytes:
comm = edge_tts.Communicate(text, EDGE_TTS_VOICE, rate=EDGE_TTS_RATE)
buf = bytearray()
async for chunk in comm.stream():
if chunk.get("type") == "audio":
buf.extend(chunk["data"])
return bytes(buf)
mp3 = asyncio.run(_gen())
if not mp3:
print("[bridge] edge TTS returned no audio", flush=True)
return None
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as t:
out_path = t.name
try:
proc = subprocess.run(
["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-i", "pipe:0", "-ac", "1", "-ar", "24000",
"-acodec", "pcm_s16le", out_path],
input=mp3, capture_output=True,
)
if proc.returncode != 0:
print(f"[bridge] edge ffmpeg transcode failed: {proc.stderr.decode('utf-8','ignore')[:200]}", flush=True)
return None
with open(out_path, "rb") as f:
return f.read()
finally:
try:
os.unlink(out_path)
except OSError:
pass
except Exception as e: # pragma: no cover - network / dep dependent
print(f"[bridge] edge synth failed: {e}", flush=True)
return None
def _melo_synthesize(text: str) -> Optional[bytes]: def _melo_synthesize(text: str) -> Optional[bytes]:
"""Synthesise via the warm MeloTTS worker (separate /opt/melo venv, Korean """Synthesise via the warm MeloTTS worker (separate /opt/melo venv, Korean
speaker @ speed 1.5). Returns a 16-bit PCM WAV, or None on any failure so speaker @ speed 1.5). Returns a 16-bit PCM WAV, or None on any failure so
@@ -356,20 +414,22 @@ def _tts_ready() -> bool:
def synthesize(text: str) -> Optional[bytes]: def synthesize(text: str) -> Optional[bytes]:
"""Synthesize text to a 16-bit PCM WAV. The primary voice is MeloTTS """Synthesize text to a 16-bit PCM WAV. The primary voice is Edge TTS (a
(Korean speaker, speed 1.5) served by the warm melo worker; Piper is a natural Korean neural voice); "melo" uses the warm MeloTTS worker. For a
fallback if the worker is unavailable. Returns None if TTS is off.""" neural engine, Piper (English) is only used if explicitly enabled, since
speaking Korean through an English voice mangles it. Returns None if off."""
if not TTS_ENABLED or not text.strip(): if not TTS_ENABLED or not text.strip():
return None return None
if TTS_ENGINE == "melo": _neural = {"edge": _edge_synthesize, "melo": _melo_synthesize}.get(TTS_ENGINE)
audio = _melo_synthesize(text) if _neural is not None:
audio = _neural(text)
if audio: if audio:
return audio return audio
if not MELO_FALLBACK_PIPER: if not MELO_FALLBACK_PIPER:
# Melo-only: better silent than mangled English for Korean text. # Neural-only: better silent than mangled English for Korean text.
print("[bridge] melo synth failed; no audio (Piper fallback disabled)", flush=True) print(f"[bridge] {TTS_ENGINE} synth failed; no audio (Piper fallback disabled)", flush=True)
return None return None
print("[bridge] melo synth failed; falling back to Piper", flush=True) print(f"[bridge] {TTS_ENGINE} synth failed; falling back to Piper", flush=True)
return _piper_synthesize(text) return _piper_synthesize(text)

View File

@@ -22,8 +22,7 @@ from typing import Any, Dict
FIELDS = [ FIELDS = [
("ollama_chat_model", "LLM 모델", "model"), ("ollama_chat_model", "LLM 모델", "model"),
("whisper_model", "STT(Whisper) 모델", "select:tiny,base,small,medium,large,large-v3"), ("whisper_model", "STT(Whisper) 모델", "select:tiny,base,small,medium,large,large-v3"),
("tts_engine", "TTS 엔진", "select:melo,piper"), ("tts_engine", "TTS 엔진", "select:edge,piper"),
("melo_speed", "TTS 속도 (MeloTTS)", "number:0.5:2.5:0.1"),
("output_language", "출력 언어 (비우면 사용자 언어)", "text"), ("output_language", "출력 언어 (비우면 사용자 언어)", "text"),
("llm_thinking_enabled", "LLM 사고(thinking) 모드", "bool"), ("llm_thinking_enabled", "LLM 사고(thinking) 모드", "bool"),
("agentic_max_turns", "에이전트 최대 반복", "number:1:12:1"), ("agentic_max_turns", "에이전트 최대 반복", "number:1:12:1"),
@@ -54,9 +53,7 @@ def _current() -> Dict[str, Any]:
cfg = _read_config() cfg = _read_config()
out: Dict[str, Any] = {} out: Dict[str, Any] = {}
for k in _KEYS: for k in _KEYS:
if k == "melo_speed": if k == "output_language":
out[k] = cfg.get("melo_speed", os.environ.get("MELO_SPEED", "1.5"))
elif k == "output_language":
out[k] = cfg.get("output_language", os.environ.get("OUTPUT_LANGUAGE", "")) out[k] = cfg.get("output_language", os.environ.get("OUTPUT_LANGUAGE", ""))
else: else:
out[k] = cfg.get(k, "") out[k] = cfg.get(k, "")
@@ -78,12 +75,7 @@ def _coerce(updates: Dict[str, Any]) -> Dict[str, Any]:
for k, v in updates.items(): for k, v in updates.items():
if k not in _KEYS: if k not in _KEYS:
continue continue
if k == "melo_speed": if k == "agentic_max_turns":
try:
v = float(v)
except (TypeError, ValueError):
continue
elif k == "agentic_max_turns":
try: try:
v = int(v) v = int(v)
except (TypeError, ValueError): except (TypeError, ValueError):
@@ -114,15 +106,15 @@ def _save(updates: Dict[str, Any]) -> None:
def _apply() -> str: def _apply() -> str:
# Restart melo + bridge AFTER this response is sent. Detached (new session) # Restart the bridge AFTER this response is sent. Detached (new session) so
# so the bridge being killed mid-restart doesn't drop the restart itself, # the bridge being killed mid-restart doesn't drop the restart itself, and
# and the HTTP client still receives this response. # the HTTP client still receives this response. (Edge TTS has no worker.)
try: try:
subprocess.Popen( subprocess.Popen(
["sh", "-c", "sleep 1; supervisorctl restart melo-worker bridge"], ["sh", "-c", "sleep 1; supervisorctl restart bridge"],
start_new_session=True, start_new_session=True,
) )
return "1초 후 브리지/TTS 워커가 재시작되어 반영됩니다." return "1초 후 브리지가 재시작되어 반영됩니다."
except Exception as e: # pragma: no cover except Exception as e: # pragma: no cover
return str(e) return str(e)

View File

@@ -5,8 +5,9 @@
# #
# docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d # docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d
# #
# Or set COMPOSE_FILE in .env (recommended): # Or set COMPOSE_FILE in .env (note the ";" separator on Windows — ":" collides
# COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-windows.yml # with the C: drive letter and breaks file resolution):
# COMPOSE_FILE=docker-compose.yml;docker-compose.gpu-windows.yml
services: services:
ollama: ollama:
deploy: deploy:

View File

@@ -66,9 +66,15 @@ services:
WHISPER_MODEL: ${WHISPER_MODEL:-medium} WHISPER_MODEL: ${WHISPER_MODEL:-medium}
WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda} WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda}
WHISPER_COMPUTE_TYPE: ${WHISPER_COMPUTE_TYPE:-float16} WHISPER_COMPUTE_TYPE: ${WHISPER_COMPUTE_TYPE:-float16}
# MeloTTS on the GPU (cu128 torch baked by docker/setup-melo.sh). CPU synth # TTS engine. Rendered into /app/config/jarvis.json via envsubst (the
# serialised under load and pushed TTS to 7-8s; GPU does ~0.3s/sentence. # bridge reads that JSON BEFORE the env, so it must carry the real engine,
MELO_DEVICE: ${MELO_DEVICE:-cuda} # not a hardcoded one — otherwise Korean text is read by the English Piper
# voice). Default edge; .env can override (e.g. piper for offline).
TTS_ENGINE: ${TTS_ENGINE:-edge}
# Edge TTS voice + rate (the chosen natural Korean voice). NOTE: edge is an
# ONLINE engine — reply text is sent to Microsoft and needs internet.
EDGE_TTS_VOICE: ${EDGE_TTS_VOICE:-ko-KR-HyunsuMultilingualNeural}
EDGE_TTS_RATE: ${EDGE_TTS_RATE:-+45%}
# Optional single-language lock for replies (empty = user's own language). # Optional single-language lock for replies (empty = user's own language).
OUTPUT_LANGUAGE: ${OUTPUT_LANGUAGE:-ko} OUTPUT_LANGUAGE: ${OUTPUT_LANGUAGE:-ko}
# Drop the pre-loop planner LLM call to cut voice-reply latency on small # Drop the pre-loop planner LLM call to cut voice-reply latency on small
@@ -97,6 +103,10 @@ services:
BROWSER_CONTROL_BIND: ${BROWSER_CONTROL_BIND:-0.0.0.0} BROWSER_CONTROL_BIND: ${BROWSER_CONTROL_BIND:-0.0.0.0}
BROWSER_CONTROL_PORT: ${BROWSER_CONTROL_PORT:-8777} BROWSER_CONTROL_PORT: ${BROWSER_CONTROL_PORT:-8777}
BROWSER_CONTROL_URL: ${BROWSER_CONTROL_URL:-} BROWSER_CONTROL_URL: ${BROWSER_CONTROL_URL:-}
# Folder of operator *.md instruction files appended to the main reply
# LLM's system prompt. Bind-mounted from ./agents below; override only to
# relocate it inside the container.
AGENTS_DIR: ${AGENTS_DIR:-/app/agents}
# No hard depends_on ollama: a browser-host (`docker compose up -d javis`) # No hard depends_on ollama: a browser-host (`docker compose up -d javis`)
# must NOT pull in Ollama. Full/bot layouts start it with a plain # must NOT pull in Ollama. Full/bot layouts start it with a plain
# `docker compose up -d` (all services); the bridge tolerates Ollama warming # `docker compose up -d` (all services); the bridge tolerates Ollama warming
@@ -149,6 +159,11 @@ services:
# If unseeded, the path fail-opens to the DDG/Brave cascade and the # If unseeded, the path fail-opens to the DDG/Brave cascade and the
# entrypoint logs a warning. Only consumed when GEMINI_AUTH=oauth. # entrypoint logs a warning. Only consumed when GEMINI_AUTH=oauth.
- ${GEMINI_OAUTH_DIR:-./docker/gemini-oauth}:/root/.gemini - ${GEMINI_OAUTH_DIR:-./docker/gemini-oauth}:/root/.gemini
# Operator instruction files. Every *.md here is appended to the main
# reply LLM's system prompt (filename order), read per turn so edits apply
# on the next reply without a rebuild/restart. Read-only; a project-
# relative path resolves identically on Linux and Windows Docker Desktop.
- ./agents:/app/agents:ro
volumes: volumes:
ollama_models: ollama_models:

View File

@@ -51,12 +51,18 @@ export JARVIS_CONFIG_PATH=/app/config/jarvis.json
# the env-rendered config, so changes survive container recreate. # the env-rendered config, so changes survive container recreate.
if [ -f /data/jarvis-settings.json ]; then if [ -f /data/jarvis-settings.json ]; then
python3 - <<'PY' || true python3 - <<'PY' || true
import json import json, os
try: try:
base = json.load(open("/app/config/jarvis.json")) base = json.load(open("/app/config/jarvis.json"))
ov = json.load(open("/data/jarvis-settings.json")) ov = json.load(open("/data/jarvis-settings.json"))
if isinstance(base, dict) and isinstance(ov, dict): if isinstance(base, dict) and isinstance(ov, dict):
base.update(ov) base.update(ov)
# A stale persisted tts_engine from an earlier voice (melo/xtts, no
# longer built into the image) would override the configured engine and
# leave the bot silent. Reset those to the env-configured engine.
if base.get("tts_engine") in ("melo", "xtts"):
base["tts_engine"] = os.environ.get("TTS_ENGINE", "edge")
print(f"[entrypoint] reset stale tts_engine -> {base['tts_engine']}")
json.dump(base, open("/app/config/jarvis.json", "w"), ensure_ascii=False, indent=2) json.dump(base, open("/app/config/jarvis.json", "w"), ensure_ascii=False, indent=2)
print("[entrypoint] merged persistent settings overrides") print("[entrypoint] merged persistent settings overrides")
except Exception as e: except Exception as e:

View File

@@ -6,7 +6,7 @@
"ollama_chat_model": "${OLLAMA_CHAT_MODEL}", "ollama_chat_model": "${OLLAMA_CHAT_MODEL}",
"intent_judge_model": "${OLLAMA_INTENT_MODEL}", "intent_judge_model": "${OLLAMA_INTENT_MODEL}",
"tts_enabled": true, "tts_enabled": true,
"tts_engine": "piper", "tts_engine": "${TTS_ENGINE}",
"tts_piper_model_path": "${TTS_PIPER_MODEL_PATH}", "tts_piper_model_path": "${TTS_PIPER_MODEL_PATH}",
"whisper_model": "${WHISPER_MODEL}", "whisper_model": "${WHISPER_MODEL}",
"whisper_backend": "faster-whisper", "whisper_backend": "faster-whisper",

View File

@@ -49,28 +49,8 @@ stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0 stderr_logfile_maxbytes=0
[program:melo-worker] # (No TTS worker program: the default Edge TTS engine synthesises in-process in
; Warm MeloTTS Korean voice (speed 1.5) in its own py3.11 venv. The bridge's # the bridge via the `edge-tts` package — no warm model/worker is needed.)
; synthesize() POSTs here; if this is down the bridge falls back to Piper.
command=/app/docker/run-if-role.sh full,bot /opt/melo/bin/python /app/bridge/melo_worker.py
directory=/app
; HF_HOME points at the dedicated, image-baked melo cache (warmed in
; setup-melo.sh). The brain's whisper_cache volume is mounted over
; /root/.cache/huggingface, so without this the pre-cached BERT + KR checkpoint
; would be shadowed and re-downloaded (and would fail if the host is offline).
; HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE force pure-cache reads: the pinned old
; transformers/huggingface_hub otherwise retry the network on every load and
; error out instead of falling back to the (complete) baked cache.
; MELO_DEVICE inherits from the container env (compose sets it; default cuda)
; so the worker runs MeloTTS on the GPU. supervisord interpolates %(ENV_x)s
; from its own environment, which is the container's.
environment=MELO_LANGUAGE="KR",MELO_SPEED="1.5",MELO_DEVICE="%(ENV_MELO_DEVICE)s",MELO_WORKER_HOST="127.0.0.1",MELO_WORKER_PORT="8770",HF_HOME="/opt/melo-cache",HF_HUB_OFFLINE="1",TRANSFORMERS_OFFLINE="1"
priority=280
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:bridge] [program:bridge]
command=/app/docker/run-if-role.sh full,bot /opt/venv/bin/python -m bridge.server command=/app/docker/run-if-role.sh full,bot /opt/venv/bin/python -m bridge.server

View File

@@ -3,6 +3,12 @@
One image, three roles (`JARVIS_ROLE`), selected in `.env`. GPU is added per OS One image, three roles (`JARVIS_ROLE`), selected in `.env`. GPU is added per OS
via a compose override picked with `COMPOSE_FILE`. via a compose override picked with `COMPOSE_FILE`.
> `COMPOSE_FILE`'s file separator is OS-specific: Linux/macOS use `:`, Windows
> uses `;` (a colon collides with the `C:` drive letter). Using `:` on Windows
> yields `... The system cannot find the file specified`. If in doubt, leave
> `COMPOSE_FILE` unset and pass the files explicitly:
> `docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d`.
## A. All-in-one (single machine) ## A. All-in-one (single machine)
Everything (desktop + Chrome + bridge + bot + TTS) in one container. Everything (desktop + Chrome + bridge + bot + TTS) in one container.
@@ -10,8 +16,8 @@ Everything (desktop + Chrome + bridge + bot + TTS) in one container.
``` ```
# .env # .env
JARVIS_ROLE=full JARVIS_ROLE=full
COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml # Ubuntu COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml # Ubuntu/macOS (":" )
# COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-windows.yml # Windows 11 # COMPOSE_FILE=docker-compose.yml;docker-compose.gpu-windows.yml # Windows 11 (";" )
DISCORD_SELFBOT_TOKEN=... DISCORD_SELFBOT_TOKEN=...
DISCORD_GUILD_ID=... DISCORD_GUILD_ID=...
@@ -45,8 +51,8 @@ Watch it on this machines VNC (`localhost:5901`) / noVNC (`localhost:6080`).
# .env # .env
JARVIS_ROLE=bot JARVIS_ROLE=bot
BROWSER_CONTROL_URL=http://192.168.10.9:8777 # the browser host's LAN IP BROWSER_CONTROL_URL=http://192.168.10.9:8777 # the browser host's LAN IP
COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml # Ubuntu COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml # Ubuntu/macOS (":" )
# COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-windows.yml # Windows 11 # COMPOSE_FILE=docker-compose.yml;docker-compose.gpu-windows.yml # Windows 11 (";" )
DISCORD_SELFBOT_TOKEN=... DISCORD_SELFBOT_TOKEN=...
DISCORD_GUILD_ID=... DISCORD_GUILD_ID=...
@@ -68,6 +74,11 @@ human-style input (visible on its VNC).
compose runs from PowerShell/cmd). Seed it once from a machine with a browser and compose runs from PowerShell/cmd). Seed it once from a machine with a browser and
the logged-in Gemini CLI (`npm i -g @google/gemini-cli`, then `gemini` -> the logged-in Gemini CLI (`npm i -g @google/gemini-cli`, then `gemini` ->
"Sign in with Google"), copying the login state: "Sign in with Google"), copying the login state:
(Note: as of 2026-06 Google blocks personal Google accounts on this CLI login
with "This client is no longer supported for Gemini Code Assist for
individuals". Workspace/org accounts may still work; personal accounts should
use `GEMINI_AUTH=apikey` with a key from https://aistudio.google.com/app/apikey
instead. Real-time search fail-opens to DDG/Brave/Wikipedia either way.)
`cp -r ~/.gemini/. docker/gemini-oauth/`. The essential file is `oauth_creds.json` `cp -r ~/.gemini/. docker/gemini-oauth/`. The essential file is `oauth_creds.json`
(it holds the refresh token; `GOOGLE_GENAI_USE_GCA=true` forces OAuth, so that is (it holds the refresh token; `GOOGLE_GENAI_USE_GCA=true` forces OAuth, so that is
the file the startup readiness check looks for) - copying the whole dir simply also the file the startup readiness check looks for) - copying the whole dir simply also

View File

@@ -13,6 +13,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- Redacted user query - Redacted user query
- Recent dialogue (last 5 minutes), including in-loop tool-call + tool-role messages from prior replies within the active conversation (tool carryover, `DialogueMemory.record_tool_turn` / `get_recent_turns_with_tools` in [src/jarvis/memory/conversation.py](src/jarvis/memory/conversation.py); per-prompt cap via `cfg.tool_carryover_max_turns` / `tool_carryover_per_entry_chars`; storage cap `_tool_turns_max_storage = 16`; cleared on `stop` signal AND on new-conversation entry; UNTRUSTED WEB EXTRACT fence markers preserved on truncation; both `content` and `tool_calls[*].function.arguments` scrubbed on write) - Recent dialogue (last 5 minutes), including in-loop tool-call + tool-role messages from prior replies within the active conversation (tool carryover, `DialogueMemory.record_tool_turn` / `get_recent_turns_with_tools` in [src/jarvis/memory/conversation.py](src/jarvis/memory/conversation.py); per-prompt cap via `cfg.tool_carryover_max_turns` / `tool_carryover_per_entry_chars`; storage cap `_tool_turns_max_storage = 16`; cleared on `stop` signal AND on new-conversation entry; UNTRUSTED WEB EXTRACT fence markers preserved on truncation; both `content` and `tool_calls[*].function.arguments` scrubbed on write)
- Unified system prompt from [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) + ASR note + tool-protocol guidance. Reply language is resolved by `reply_language_directive(lang, cfg.tts_engine)` where `lang = _resolve_output_language()` — the single source of truth that prefers the settings-web UI value (config JSON `output_language`) over the compose `OUTPUT_LANGUAGE` env, so changing the language in the settings page takes effect. An explicit lock wins (forces "reply only in `<language>`", also forbidding other scripts so small models stop leaking trailing CJK/Hanja); else a Piper/Chatterbox TTS forces English (English-only voices); else (multilingual TTS, no lock) the assistant replies in the user's own language. The directive is inserted near the FRONT of the guidance list so a small model gives it primacy, and the SAME resolved `lang` feeds `build_system_prompt()`, which rewrites the persona's "in the user's language" clause to the locked language so the persona cannot contradict the directive (previously the persona read the raw env while the directive read the config value, so a settings-UI change was honoured by one and ignored by the other). Gated in `_build_initial_system_message()` at [engine.py](src/jarvis/reply/engine.py). - Unified system prompt from [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) + ASR note + tool-protocol guidance. Reply language is resolved by `reply_language_directive(lang, cfg.tts_engine)` where `lang = _resolve_output_language()` — the single source of truth that prefers the settings-web UI value (config JSON `output_language`) over the compose `OUTPUT_LANGUAGE` env, so changing the language in the settings page takes effect. An explicit lock wins (forces "reply only in `<language>`", also forbidding other scripts so small models stop leaking trailing CJK/Hanja); else a Piper/Chatterbox TTS forces English (English-only voices); else (multilingual TTS, no lock) the assistant replies in the user's own language. The directive is inserted near the FRONT of the guidance list so a small model gives it primacy, and the SAME resolved `lang` feeds `build_system_prompt()`, which rewrites the persona's "in the user's language" clause to the locked language so the persona cannot contradict the directive (previously the persona read the raw env while the directive read the config value, so a settings-UI change was honoured by one and ignored by the other). Gated in `_build_initial_system_message()` at [engine.py](src/jarvis/reply/engine.py).
- **Operator instructions** (two sources, both framed "Additional instructions from the operator:" and appended near the end of the guidance list): the settings-UI `llm_instructions` config field, and every `*.md` file in `AGENTS_DIR` (default `/app/agents`, bind-mounted from `./agents`). The file-based set is read once per turn by `load_agent_instructions()` in [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) and concatenated in filename order, so dropping/editing a `.md` applies on the next reply with no rebuild/restart; fail-open to `""` when the folder is absent/empty/unreadable.
- **Warm profile block** (query-agnostic User + Directives excerpt from the knowledge graph, composed by `build_warm_profile()` / `format_warm_profile_block()` in [src/jarvis/memory/graph_ops.py](src/jarvis/memory/graph_ops.py) at Step 3.5 of `reply()`; no LLM call, pure SQLite read; injected unconditionally so personalisation is the default; result cached in `DialogueMemory._hot_cache` under `DialogueMemory.WARM_PROFILE_CACHE_KEY` for the lifetime of the active conversation. Invalidated on `stop`, on new-conversation entry, AND on User/Directives graph mutations via the listener registered in [src/jarvis/daemon.py](src/jarvis/daemon.py) against `register_graph_mutation_listener` in [src/jarvis/memory/graph.py](src/jarvis/memory/graph.py); World-branch writes are ignored) - **Warm profile block** (query-agnostic User + Directives excerpt from the knowledge graph, composed by `build_warm_profile()` / `format_warm_profile_block()` in [src/jarvis/memory/graph_ops.py](src/jarvis/memory/graph_ops.py) at Step 3.5 of `reply()`; no LLM call, pure SQLite read; injected unconditionally so personalisation is the default; result cached in `DialogueMemory._hot_cache` under `DialogueMemory.WARM_PROFILE_CACHE_KEY` for the lifetime of the active conversation. Invalidated on `stop`, on new-conversation entry, AND on User/Directives graph mutations via the listener registered in [src/jarvis/daemon.py](src/jarvis/daemon.py) against `register_graph_mutation_listener` in [src/jarvis/memory/graph.py](src/jarvis/memory/graph.py); World-branch writes are ignored)
- Digested memory enrichment (optional, see #4) - Digested memory enrichment (optional, see #4)
- Time + location context (re-injected each turn) - Time + location context (re-injected each turn)

View File

@@ -608,7 +608,11 @@ def load_settings() -> Settings:
active_profiles = _ensure_list(merged.get("active_profiles")) active_profiles = _ensure_list(merged.get("active_profiles"))
tts_enabled = bool(merged.get("tts_enabled", True)) tts_enabled = bool(merged.get("tts_enabled", True))
tts_engine = str(merged.get("tts_engine", "piper")).lower() tts_engine = str(merged.get("tts_engine", "piper")).lower()
if tts_engine not in ("piper", "chatterbox"): # "edge" (Microsoft Edge TTS) is the containerized bridge's Korean voice;
# "melo" is the legacy warm-worker voice. Both are multilingual, so they must
# be preserved here — coercing them to "piper" would mislabel the engine as
# English-only in reply_language_directive().
if tts_engine not in ("piper", "chatterbox", "edge", "melo"):
tts_engine = "piper" # Default to piper if invalid value tts_engine = "piper" # Default to piper if invalid value
tts_voice_val = merged.get("tts_voice") tts_voice_val = merged.get("tts_voice")
tts_voice = None if tts_voice_val in (None, "", "null") else str(tts_voice_val) tts_voice = None if tts_voice_val in (None, "", "null") else str(tts_voice_val)

View File

@@ -9,7 +9,11 @@ import os
from typing import Optional, TYPE_CHECKING from typing import Optional, TYPE_CHECKING
from ..utils.redact import redact from ..utils.redact import redact
from ..system_prompt import build_system_prompt, reply_language_directive from ..system_prompt import (
build_system_prompt,
load_agent_instructions,
reply_language_directive,
)
from ..tools.registry import run_tool_with_retries, generate_tools_description, generate_tools_json_schema, BUILTIN_TOOLS from ..tools.registry import run_tool_with_retries, generate_tools_description, generate_tools_json_schema, BUILTIN_TOOLS
from ..tools.builtin.stop import STOP_SIGNAL from ..tools.builtin.stop import STOP_SIGNAL
from ..debug import debug_log from ..debug import debug_log
@@ -1702,6 +1706,10 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
# the directive used the config value made the two contradict each other. # the directive used the config value made the two contradict each other.
_output_language = _resolve_output_language() _output_language = _resolve_output_language()
_persona_prompt = build_system_prompt(_assistant_name, _output_language) _persona_prompt = build_system_prompt(_assistant_name, _output_language)
# File-based operator instructions: every *.md in AGENTS_DIR (default
# /app/agents, bind-mounted from ./agents). Read once per turn so edits in
# the folder apply on the next reply without a restart; fail-open to "".
_agent_instructions = load_agent_instructions()
def _build_initial_system_message() -> str: def _build_initial_system_message() -> str:
guidance = [_persona_prompt.strip()] guidance = [_persona_prompt.strip()]
@@ -1810,6 +1818,12 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
if _user_instructions: if _user_instructions:
guidance.append("Additional instructions from the operator:\n" + _user_instructions) guidance.append("Additional instructions from the operator:\n" + _user_instructions)
# File-based operator instructions: the concatenated agents/*.md content
# resolved once above. Same framing/placement as the settings-UI field
# so both are treated as authoritative operator guidance.
if _agent_instructions:
guidance.append("Additional instructions from the operator:\n" + _agent_instructions)
# Recency reinforcement: repeat the language lock at the very END too. # Recency reinforcement: repeat the language lock at the very END too.
# In a ~5k-token prompt the front-placed rule gets "lost in the middle"; # In a ~5k-token prompt the front-placed rule gets "lost in the middle";
# bigger models (qwen2.5:7b) otherwise leak Chinese/Cyrillic mid-reply. # bigger models (qwen2.5:7b) otherwise leak Chinese/Cyrillic mid-reply.

View File

@@ -6,8 +6,51 @@ who renames the wake word (e.g. "Friday") gets a butler with the matching
name rather than a persona hardcoded to "Jarvis". name rather than a persona hardcoded to "Jarvis".
""" """
import os
from pathlib import Path
from typing import Optional from typing import Optional
# Default location of the operator's file-based instruction folder. In the
# Docker deployment ./agents is bind-mounted here (see docker-compose.yml), so a
# user can drop *.md files in without rebuilding. Overridable via AGENTS_DIR.
_DEFAULT_AGENTS_DIR = "/app/agents"
def load_agent_instructions(agents_dir: Optional[str] = None) -> str:
"""Concatenate every ``*.md`` in the agents dir into one instruction block.
Files are read in filename order (so ``00-tone.md`` precedes ``10-rules.md``)
and joined with blank lines. This lets the operator extend the main reply
LLM's system prompt by dropping Markdown files into a folder, no code change
or restart required — the caller reads this once per turn.
Resolution order for the directory: explicit ``agents_dir`` arg, then the
``AGENTS_DIR`` env var, then ``/app/agents``.
Fail-open by design: a missing or empty directory, an unreadable file, or
any unexpected error yields ``""`` so a misconfigured folder can never break
a reply. Only regular ``*.md`` files are read; other files are ignored.
"""
directory = agents_dir or os.environ.get("AGENTS_DIR") or _DEFAULT_AGENTS_DIR
try:
base = Path(directory)
if not base.is_dir():
return ""
parts: list[str] = []
for path in sorted(base.glob("*.md"), key=lambda p: p.name):
if not path.is_file():
continue
try:
text = path.read_text(encoding="utf-8").strip()
except Exception:
continue
if text:
parts.append(text)
return "\n\n".join(parts).strip()
except Exception:
return ""
_SYSTEM_PROMPT_TEMPLATE: str = ( _SYSTEM_PROMPT_TEMPLATE: str = (
"Persona: you are a British butler named {name} — polite, composed, quietly amused, and " "Persona: you are a British butler named {name} — polite, composed, quietly amused, and "
"quietly enjoying yourself. Default voice is dry, witty, and lightly sarcastic: you notice " "quietly enjoying yourself. Default voice is dry, witty, and lightly sarcastic: you notice "

View File

@@ -22,7 +22,29 @@ path (evals, text entry) and falls back to the master flag:
- **on-screen Chrome**: `browser_search()` drives Chrome (Node CDP helper - **on-screen Chrome**: `browser_search()` drives Chrome (Node CDP helper
`bot/scripts/stream-test/browse-search.mjs`) to Google-search the query, so `bot/scripts/stream-test/browse-search.mjs`) to Google-search the query, so
the action is visible on the Go-Live broadcast. the action is visible on the Go-Live broadcast. The helper searches the
human way — it loads the site home page, types the query into the search box
one key at a time, and presses Enter (both Google `search` and `youtube`),
rather than jumping to a results URL. When no broadcast Chrome is
reachable on CDP (e.g. a plain text turn with no active broadcast), the helper
falls back, for `search` only, to launching its own Chrome so browser-based
Google search still works with no API cost. Fallback order:
- **CDP** (the broadcast Chrome) — preferred, visible on the stream.
- **Persistent profile** when `CHROME_USER_DATA_DIR` is set — Chrome opened
against that profile dir (system `channel: 'chrome'`, else bundled chromium).
Logging that dedicated profile into Google once lets Google treat later
searches as a returning signed-in user, which is what avoids the
bot-detection interstitial. This is the reliable way to get browser Google
search in plain text turns.
- **Ephemeral headless** otherwise — a fresh anonymous session; works only
where Google does not challenge it (e.g. a non-flagged residential IP).
The `youtube` action never uses the fallback (it only makes sense on the
visible broadcast Chrome). Caveat: an anonymous (not-signed-in) session can be
served Google's bot-detection interstitial (`/sorry/index`); the helper
detects this structurally by URL and fails fast, so the caller fail-opens to
the DDG / Brave / Wikipedia cascade rather than treating the challenge page as
"no results".
- **Gemini**: answers, with the sub-mode chosen by `cfg.gemini_auth` - **Gemini**: answers, with the sub-mode chosen by `cfg.gemini_auth`
(env `GEMINI_AUTH`, default `oauth`): (env `GEMINI_AUTH`, default `oauth`):
- `oauth` (default): `gemini_cli_search()` shells out to the Gemini CLI - `oauth` (default): `gemini_cli_search()` shells out to the Gemini CLI

View File

@@ -7,6 +7,7 @@ hardcoded to Jarvis.
from jarvis.system_prompt import ( from jarvis.system_prompt import (
build_system_prompt, build_system_prompt,
load_agent_instructions,
output_language_directive, output_language_directive,
reply_language_directive, reply_language_directive,
ENGLISH_ONLY_DIRECTIVE, ENGLISH_ONLY_DIRECTIVE,
@@ -108,3 +109,65 @@ class TestReplyLanguageDirective:
def test_lock_wins_even_with_multilingual_tts(self): def test_lock_wins_even_with_multilingual_tts(self):
directive = reply_language_directive("Korean", "melo") directive = reply_language_directive("Korean", "melo")
assert directive is not None and "Korean" in directive assert directive is not None and "Korean" in directive
def test_edge_is_multilingual(self):
# Edge TTS (the default Korean voice) is not English-only: no lock → the
# user's own language, and a lock is honoured (not forced to English).
assert reply_language_directive(None, "edge") is None
directive = reply_language_directive("Korean", "edge")
assert directive is not None and "Korean" in directive
assert directive != ENGLISH_ONLY_DIRECTIVE
class TestLoadAgentInstructions:
"""Operator can extend the reply LLM's system prompt by dropping *.md files
into an agents/ folder. The loader concatenates them in filename order and
fails open so a missing/empty/broken folder never breaks a reply."""
def test_missing_dir_returns_empty(self, tmp_path):
assert load_agent_instructions(str(tmp_path / "does-not-exist")) == ""
def test_empty_dir_returns_empty(self, tmp_path):
assert load_agent_instructions(str(tmp_path)) == ""
def test_reads_and_concatenates_single_file(self, tmp_path):
(tmp_path / "rules.md").write_text("Always be brief.", encoding="utf-8")
assert load_agent_instructions(str(tmp_path)) == "Always be brief."
def test_files_are_ordered_by_filename(self, tmp_path):
# Filename prefixes let the operator control ordering.
(tmp_path / "10-second.md").write_text("SECOND", encoding="utf-8")
(tmp_path / "00-first.md").write_text("FIRST", encoding="utf-8")
result = load_agent_instructions(str(tmp_path))
assert result.index("FIRST") < result.index("SECOND")
def test_only_md_files_are_read(self, tmp_path):
(tmp_path / "note.txt").write_text("IGNORE ME", encoding="utf-8")
(tmp_path / "use.md").write_text("USE ME", encoding="utf-8")
result = load_agent_instructions(str(tmp_path))
assert "USE ME" in result
assert "IGNORE ME" not in result
def test_blank_files_are_skipped(self, tmp_path):
(tmp_path / "blank.md").write_text(" \n ", encoding="utf-8")
(tmp_path / "real.md").write_text("Real instruction.", encoding="utf-8")
assert load_agent_instructions(str(tmp_path)) == "Real instruction."
def test_env_var_is_used_when_no_arg(self, tmp_path, monkeypatch):
(tmp_path / "a.md").write_text("FROM ENV", encoding="utf-8")
monkeypatch.setenv("AGENTS_DIR", str(tmp_path))
assert load_agent_instructions() == "FROM ENV"
def test_explicit_arg_overrides_env(self, tmp_path, monkeypatch):
(tmp_path / "env.md").write_text("ENV", encoding="utf-8")
other = tmp_path / "other"
other.mkdir()
(other / "arg.md").write_text("ARG", encoding="utf-8")
monkeypatch.setenv("AGENTS_DIR", str(tmp_path))
assert load_agent_instructions(str(other)) == "ARG"
def test_a_file_path_instead_of_dir_returns_empty(self, tmp_path):
f = tmp_path / "file.md"
f.write_text("x", encoding="utf-8")
# Pointed at a file, not a directory → fail-open.
assert load_agent_instructions(str(f)) == ""

View File

@@ -0,0 +1,35 @@
"""The container's TTS engine must be env-driven, not hardcoded.
Regression for a bug where docker/jarvis-config.template.json hardcoded
`"tts_engine": "piper"`. The bridge reads the rendered /app/config/jarvis.json
*before* the environment, so a hardcoded "piper" overrode `TTS_ENGINE=melo` in
.env and the bot read Korean text with the English Piper voice ("foreign
accent"). The template must carry `${TTS_ENGINE}` so envsubst (entrypoint.sh)
renders whatever engine the deployment configured.
"""
import json
import string
from pathlib import Path
TEMPLATE = Path(__file__).resolve().parent.parent / "docker" / "jarvis-config.template.json"
def _render(**env) -> dict:
"""Mimic entrypoint.sh `envsubst < template`: substitute env vars, leaving
any unset ones as literal text (valid JSON string values)."""
raw = TEMPLATE.read_text(encoding="utf-8")
return json.loads(string.Template(raw).safe_substitute(**env))
def test_template_does_not_hardcode_an_engine():
raw = TEMPLATE.read_text(encoding="utf-8")
assert '"tts_engine": "${TTS_ENGINE}"' in raw
assert '"tts_engine": "piper"' not in raw
assert '"tts_engine": "melo"' not in raw
def test_rendered_engine_follows_env():
assert _render(TTS_ENGINE="melo")["tts_engine"] == "melo"
assert _render(TTS_ENGINE="piper")["tts_engine"] == "piper"
assert _render(TTS_ENGINE="xtts")["tts_engine"] == "xtts"