diff --git a/docs/llm_contexts.md b/docs/llm_contexts.md index e183b0f..bf63291 100644 --- a/docs/llm_contexts.md +++ b/docs/llm_contexts.md @@ -2,7 +2,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it is gated. This is the reference for optimising the app's main bottleneck (LLM latency). Keep it in sync with the code — see the note at the bottom. -**Transport (all contexts below).** `ollama_base_url` selects the API. A URL that ends in `/v1` (v2: Strata at `http://127.0.0.1:8080/v1`; also llama.cpp or Ollama's own `/v1`) uses the OpenAI-compatible API: chat goes to `/chat/completions`, embeddings to `/embeddings`, and the warm-up is a one-token chat completion. Any other URL uses Ollama's `/api/chat`, `/api/generate` and `/api/embeddings`. The conversion lives in `src/jarvis/llm.py` only: callers keep sending Ollama-shaped messages and reading Ollama-shaped responses (`message.content`, `message.tool_calls` with dict arguments, `message.thinking` from `reasoning_content`). On the OpenAI path `num_ctx` and `keep_alive` are not sent (the server owns its context size and residency), `num_predict` becomes `max_tokens`, and the thinking switch becomes `chat_template_kwargs.enable_thinking`. The `num_ctx: 8192` limits listed below apply to the Ollama path. A server with no embedding model (Strata) answers `/embeddings` with an error and every embedding consumer fails open to its text path. +**Transport (all contexts below).** `ollama_base_url` selects the API. A URL that ends in `/v1` (v2: Strata at `http://127.0.0.1:8080/v1`; also llama.cpp or Ollama's own `/v1`) uses the OpenAI-compatible API: chat goes to `/chat/completions`, embeddings to `/embeddings`, and the warm-up is a one-token chat completion. Any other URL uses Ollama's `/api/chat`, `/api/generate` and `/api/embeddings`. The conversion lives in `src/jarvis/llm.py` only: callers keep sending Ollama-shaped messages and reading Ollama-shaped responses (`message.content`, `message.tool_calls` with dict arguments, `message.thinking` from `reasoning_content`). On the OpenAI path `num_ctx` and `keep_alive` are not sent (the server owns its context size and residency), `num_predict` becomes `max_tokens`, and the thinking switch becomes `chat_template_kwargs.enable_thinking`. The `num_ctx: 8192` limits listed below apply to the Ollama path. Ollama's own `/v1` ignores `num_ctx` and runs at its default context (2,048 tokens here), which cuts the system prompt; to test the OpenAI path on Ollama use a model tag created with a larger `num_ctx`. A server with no embedding model (Strata) answers `/embeddings` with an error and every embedding consumer fails open to its text path. --- @@ -19,7 +19,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i - **Operator instructions** (two sources, both framed "Additional instructions from the operator:" and appended near the end of the guidance list): the settings-UI `llm_instructions` config field, and every `*.md` file in `AGENTS_DIR` (default `/app/agents`, bind-mounted from `./agents`). The file-based set is read once per turn by `load_agent_instructions()` in [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) and concatenated in filename order, so dropping/editing a `.md` applies on the next reply with no rebuild/restart; fail-open to `""` when the folder is absent/empty/unreadable. - **Warm profile block** (query-agnostic User + Directives excerpt from the knowledge graph, composed by `build_warm_profile()` / `format_warm_profile_block()` in [src/jarvis/memory/graph_ops.py](src/jarvis/memory/graph_ops.py) at Step 3.5 of `reply()`; no LLM call, pure SQLite read; injected unconditionally so personalisation is the default; result cached in `DialogueMemory._hot_cache` under `DialogueMemory.WARM_PROFILE_CACHE_KEY` for the lifetime of the active conversation. Invalidated on `stop`, on new-conversation entry, AND on User/Directives graph mutations via the listener registered in [src/jarvis/daemon.py](src/jarvis/daemon.py) against `register_graph_mutation_listener` in [src/jarvis/memory/graph.py](src/jarvis/memory/graph.py); World-branch writes are ignored) - Digested memory enrichment (optional, see #4) - - **v2 history context** (optional, caller-supplied `history=` on `run_reply_engine`): the recent window and search matches from the history store ([src/jarvis/memory/history.py](src/jarvis/memory/history.py), [spec](src/jarvis/memory/history.spec.md)), scoped to the asker and capped by the store's size budget (default about 4,000 tokens, matches at most half). No LLM call; not gated by the planner. Injected in a `` fence labelled "Information shared in previous conversations"; the lines are data, not instructions. Eval: `evals/test_history_context_recall.py`. + - **v2 history context** (optional, caller-supplied `history=` on `run_reply_engine`): the recent window and search matches from the history store ([src/jarvis/memory/history.py](src/jarvis/memory/history.py), [spec](src/jarvis/memory/history.spec.md)), scoped to the asker and capped by the store's size budget (default about 4,000 tokens, matches at most half). No LLM call; not gated by the planner. Injected in a `` fence labelled "Information shared in previous conversations"; the lines are data, not instructions. Eval: `evals/test_history_context_recall.py`. With `legacy_memory=False` (the v2 brain in the voice worker) the warm profile and the whole memory enrichment chain (#3 extractor, #3b recall gate, #4 digest and the diary/graph reads) are skipped for the turn, because v1's long-term memory belongs to a single user. - Time + location context (re-injected each turn) - Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68)) - Tool results from prior turns (raw or digested — see #5) diff --git a/docs/v2/roadmap.md b/docs/v2/roadmap.md index 227315f..7134799 100644 --- a/docs/v2/roadmap.md +++ b/docs/v2/roadmap.md @@ -7,7 +7,7 @@ | M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. 가상 모니터 확인을 뺀 모든 항목 측정 완료(2026-11-10). 가상 모니터 확인은 M6에서 합니다. 2026-11-10에 추가한 `desktop` 단계(agent-desktop)는 측정을 마쳤고, 자동 성능 조절용 `gamemode` 단계는 세 번 쟀고, `gamemode_stt`(LLM과 STT를 함께 켬, STT GPU·CPU)와 `gamemode_live`(실제 게임을 켬) 단계도 측정을 마쳤습니다(2026-11-10). 결과로 게임 중 최소 배치는 전문가 2GB로 정했고, 평소 배치에 STT 자리를 남기는 것과 게임 중 LLM이 답하는 동안 부분 전사 간격을 늘리는 것(게임이 없을 때는 그대로)도 정했습니다(사용자 결정, 2026-11-10). `gamemode_tune` 단계도 측정을 마쳤고(2026-11-11), 게임 중 LLM이 답하는 동안에는 부분 전사를 멈추기로 정했습니다. GPU 95% 제한은 측정에서 사용률을 낮추지 못해 기본값을 끔으로 바꿨고(사용자 결정, 2026-11-11), `video_drops` 단계(Jarvis를 돌리는 동안 유튜브 영상의 끊긴 프레임, 직접 실행)도 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 영상이 끊기지 않았고 제한이 끊김을 줄이지 못해 설정을 뺐습니다. 대신 게임 화면의 프레임을 직접 재는 `game_frames` 단계(게임을 켠 채로 Jarvis를 끈 때·쉴 때·바쁠 때와 우선순위를 낮춘 양보 상태, 직접 실행, 관리자 권한)를 넣었고 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 게임의 평균 fps와 1% low가 5% 넘게 떨어지지 않아 양보는 넣지 않기로 했습니다. 남은 M0 측정은 가상 모니터 확인(M6)입니다. 측정 키트: [scripts/m0/](../../scripts/m0/README.md) | | M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과. 구현과 테스트 완료(2026-11-11): `src/voice_worker/`, 스펙은 [voice_worker.spec.md](../../src/voice_worker/voice_worker.spec.md). 이 서버의 RTX 5050에서 10명이 겹쳐 말하는 실시간 입력(48kHz 스테레오, M0 한국어 샘플 8개를 10명에게 나눠 줌, 절반은 디스코드 speaking 종료, 절반은 무음으로 끝남)을 넣어 10명 모두 최종 전사를 받았고 문자 오류율은 평균 4%, 최대 19%("일곱 시"를 "7시"로 적은 것), 말 끝 판정부터 최종 전사 전송까지 중앙값 0.64초, 최대 1.56초였습니다. TTS는 첫 문장 소리가 0.2초에 나왔고, 그 소리를 다시 넣으면 같은 문장으로 받아 적혔습니다. 샘플이 MeloTTS 합성 음성이라 실제 사람 녹음과 게임PC 수치는 아직 없습니다 | | M2 | 디스코드 음성 코어 이전 | v1 봇 기능을 새 앱 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기). 구현과 테스트 완료(2026-11-11): `app/`(Rust, serenity + songbird), 스펙은 [discord_core.spec.md](../../app/src/discord_core.spec.md). 처음 Electron + TypeScript로 만든 것을 사용자 요청으로 Rust로 다시 씀. 앱이 음성 워커를 띄우고(실행마다 새 토큰, 죽으면 다시 시작) 연결이 끊기면 다시 붙습니다. 음성 채널에 들어갈 때마다 전사 고지를 채널 채팅에 올립니다. songbird 0.6.1의 DAVE 수신 해독이 조각 3분의 1가량을 깨뜨려서(20ms마다 음이 바뀌는 시험 신호로 확인, 보내기는 정상) 받은 패킷의 순서 맞추기와 Opus 해독은 앱이 직접 합니다. 실제 디스코드에서 봇 두 개로 확인(이 서버의 RTX 5050): 다른 봇이 채널에 튼 녹음 문장 5개를 모두 받아 적었고(문자 오류율 3개 0%, "기록해"를 "비록해"로 적은 1개 5%, "일곱 시"를 "7시"로 적은 1개 12.5%), 말이 끝난 뒤 최종 전사까지 0.71~0.90초였습니다. 자비스가 말한 문장을 상대 봇이 받아 다시 전사하니 오류율 3.4%였고, 정지 명령 뒤 0.08초 만에 소리가 끊겼습니다. 채팅 채널은 새 글 받기, 최근 글 읽기, 보내기(2,000자 넘으면 나눔, 아무도 호출하지 않음)가 됩니다. 사람이 직접 말한 소리와 게임PC(Windows) 빌드는 아직 확인하지 않았고, 음성으로 "자비스, 멈춰"는 호출 판정이 필요해 M4에서 합니다 | -| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과. 1단계 완료(2026-11-11): LLM 주소가 `/v1`로 끝나면 OpenAI 호환 방식(`/chat/completions`, `/embeddings`)으로 부름. 형식 변환은 `src/jarvis/llm.py` 한곳에서 하고 두뇌의 나머지 코드는 그대로 씀. 대화, 도구 호출, 스트리밍, 의도 판정, 받아쓰기 정리, 미리 불러오기가 모두 이 방식으로 감. 이 서버 Ollama의 `/v1` 주소와 qwen2.5:7b로 실제 두뇌 한 턴을 돌려 "서울 날씨 어때?"에 날씨 도구를 부르고 그 결과로 답하는 것을 확인함. Strata에는 임베딩 모델이 없어 지난 기록 검색은 낱말 검색으로 함. 2단계 완료(2026-11-11): 대화 기록 저장소 `src/jarvis/memory/history.py`(스펙 [history.spec.md](../../src/jarvis/memory/history.spec.md)). 음성 전사, 자비스의 답, 채팅 채널 글, 음성 채널 출입 시간을 저장하고, 묻는 사람이 볼 수 있는 기록(본인 기록, 채팅 채널, 함께 있던 동안의 다른 사람 말)만 찾음. 전체 기간에서 관련도 순으로 찾고, 기록 10만 개에서 검색 48~143ms. 두뇌는 찾은 기록을 받아 지시문에 참고 자료로 넣음(평가 `evals/test_history_context_recall.py`, qwen2.5:7b에서 3번 모두 통과). 3단계 완료(2026-11-11): 앱이 음성 워커에 음성 채널 입장·퇴장, 사람 출입, 등록한 채팅 채널의 글·수정·삭제를 알리고, 워커가 최종 전사를 저장함. 워커는 `--history`로 저장소를, `--brain`으로 두뇌를 켜고, `reply` 요청에 묻는 사람의 범위로 찾은 기록을 넣어 답하고(`reply_text`) 음성이면 바로 말함. 실제 디스코드 시험(봇 두 개)에서 전사 8개가 이름과 함께 저장되고, 채팅 수정은 반영, 삭제한 글은 빠지고, 자비스가 나간 뒤 출입 기록이 모두 닫힘. 워커와 실제 두뇌(이 서버 Ollama `/v1`, qwen2.5:7b)로 다른 사람이 채팅에 남긴 약속을 묻자 맞게 답함. 언제 `reply`를 보낼지(호출 판정)와 음성 답 재생 연결은 M4. 남은 일: 채팅 채널 도구, 기존 평가 통과 확인 | +| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과. 1단계 완료(2026-11-11): LLM 주소가 `/v1`로 끝나면 OpenAI 호환 방식(`/chat/completions`, `/embeddings`)으로 부름. 형식 변환은 `src/jarvis/llm.py` 한곳에서 하고 두뇌의 나머지 코드는 그대로 씀. 대화, 도구 호출, 스트리밍, 의도 판정, 받아쓰기 정리, 미리 불러오기가 모두 이 방식으로 감. 이 서버 Ollama의 `/v1` 주소와 qwen2.5:7b로 실제 두뇌 한 턴을 돌려 "서울 날씨 어때?"에 날씨 도구를 부르고 그 결과로 답하는 것을 확인함. Strata에는 임베딩 모델이 없어 지난 기록 검색은 낱말 검색으로 함. 2단계 완료(2026-11-11): 대화 기록 저장소 `src/jarvis/memory/history.py`(스펙 [history.spec.md](../../src/jarvis/memory/history.spec.md)). 음성 전사, 자비스의 답, 채팅 채널 글, 음성 채널 출입 시간을 저장하고, 묻는 사람이 볼 수 있는 기록(본인 기록, 채팅 채널, 함께 있던 동안의 다른 사람 말)만 찾음. 전체 기간에서 관련도 순으로 찾고, 기록 10만 개에서 검색 48~143ms. 두뇌는 찾은 기록을 받아 지시문에 참고 자료로 넣음(평가 `evals/test_history_context_recall.py`, qwen2.5:7b에서 3번 모두 통과). 3단계 완료(2026-11-11): 앱이 음성 워커에 음성 채널 입장·퇴장, 사람 출입, 등록한 채팅 채널의 글·수정·삭제를 알리고, 워커가 최종 전사를 저장함. 워커는 `--history`로 저장소를, `--brain`으로 두뇌를 켜고, `reply` 요청에 묻는 사람의 범위로 찾은 기록을 넣어 답하고(`reply_text`) 음성이면 바로 말함. 실제 디스코드 시험(봇 두 개)에서 전사 8개가 이름과 함께 저장되고, 채팅 수정은 반영, 삭제한 글은 빠지고, 자비스가 나간 뒤 출입 기록이 모두 닫힘. 워커와 실제 두뇌(이 서버 Ollama `/v1`, qwen2.5:7b, 맥락 8천 토큰)로 다른 사람이 채팅에 남긴 약속을 4번 물어 4번 모두 맞게 답함. Ollama의 `/v1`은 맥락을 2,048토큰으로 고정해 지시문을 자르므로, Ollama로 시험할 때는 `num_ctx`를 넣은 모델 이름을 씀(Strata는 시작할 때 맥락을 정해 해당 없음). v2 두뇌는 한 사람용인 v1 장기 기억(프로필·일기·그래프)을 넣지 않고, 답하는 동안 원문이 로그에 찍히지 않게 함. 언제 `reply`를 보낼지(호출 판정)와 음성 답 재생 연결은 M4. 남은 일: 채팅 채널 도구, 기존 평가 통과 확인 | | M4 | 왕복 완성과 지연 튜닝 | 호출어·호출 방식·지정한 사람 설정, 동시 대화 응답 정책 4가지(기본 "처음 대화만")와 정책별 끼어들기 규칙. 말 끝부터 첫 소리까지 실측 1.5초 이내, 10명 채널 부하 테스트 | | M5 | 관리 앱 기본 | 첫 실행 마법사, 데이터 폴더, 런타임·모델 자동 설치, 대시보드, 봇/채널·호출/채팅 채널(디스코드 채널 이름·주제 반영 포함)/대화 기억/모델 설정, 로그 화면(보기·복사·다운로드) | | M6 | 기능 화면과 브라우저 | 기능 켜기/끄기, MCP 관리, 봇용 크롬(가상 모니터에 둠, Virtual Display Driver 포함), 예약 수집 틀(대상은 비움). 디스코드 "화면" 탭 공유, 커서, 모니터 해제 때 창 위치 확인 | diff --git a/evals/test_history_context_recall.py b/evals/test_history_context_recall.py index 6fb3ff5..38afca2 100644 --- a/evals/test_history_context_recall.py +++ b/evals/test_history_context_recall.py @@ -46,7 +46,9 @@ CASES = [ { # Field run 2026-11-11 (qwen2.5:7b through the voice worker): a chat channel record from # someone else, asked about by the user. The reply opened by reading the record line out, - # stamp and all, which TTS would speak. + # stamp and all, which TTS would speak. That run went through Ollama's /v1, which fixes the + # context at 2,048 tokens and cut the system prompt; with the full prompt (8k) the same + # question was answered in own words 4 of 4 times. Kept as a guard. "id": "korean-chat-record", "query": "이번 주 모임 언제 어디라고 했지?", "recent": [], diff --git a/src/jarvis/memory/history.py b/src/jarvis/memory/history.py index e95f702..afa6e2a 100644 --- a/src/jarvis/memory/history.py +++ b/src/jarvis/memory/history.py @@ -181,6 +181,13 @@ class HistoryStore: (ts, user_id, channel_id)) self._conn.commit() + def close_open_presence(self, ts: float) -> int: + """Close every open interval (at start, after a run that ended without closing them).""" + with self._lock: + n = self._conn.execute("UPDATE presence SET left_at=? WHERE left_at IS NULL", (ts,)).rowcount + self._conn.commit() + return n + def presence_close_channel(self, channel_id: str, ts: float) -> None: with self._lock: self._conn.execute("UPDATE presence SET left_at=? WHERE channel_id=? AND left_at IS NULL", diff --git a/src/jarvis/memory/history.spec.md b/src/jarvis/memory/history.spec.md index 554ef00..cfdaba0 100644 --- a/src/jarvis/memory/history.spec.md +++ b/src/jarvis/memory/history.spec.md @@ -71,7 +71,9 @@ Setting `since` (default none, meaning the whole history) drops records older th ## Writes and maintenance - `add_voice`, `add_reply`, `add_chat`, `edit_chat`, `delete_chat`. -- `presence_join`, `presence_leave`, `presence_close_channel`. +- `presence_join`, `presence_leave`, `presence_close_channel`, and `close_open_presence(ts)`, which the + host calls at start: a run that was killed never closed its intervals, and nobody is heard until + the app reports who is there again. - `records_of(person)` and `forget_person(person)`: the person's own records (authored by them or replies to them), for the data screen's per-person view and delete. Deleting also removes their presence intervals. diff --git a/src/jarvis/reply/engine.py b/src/jarvis/reply/engine.py index b32c6c2..8f0a02a 100644 --- a/src/jarvis/reply/engine.py +++ b/src/jarvis/reply/engine.py @@ -1023,7 +1023,8 @@ def _history_block(history: Any) -> str: def run_reply_engine(db: "Database", cfg, tts: Optional[Any], text: str, dialogue_memory: "DialogueMemory", language: Optional[str] = None, - history: Optional[Any] = None) -> Optional[str]: + history: Optional[Any] = None, + legacy_memory: bool = True) -> Optional[str]: """ Main entry point for reply generation. @@ -1042,6 +1043,10 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any], asker from the history store (recent window and search matches, already scoped to what this asker may see). Injected as a fenced reference block; None or empty adds nothing. + legacy_memory: v1's long-term memory (warm profile, diary and graph + enrichment) belongs to a single user. False skips all of it, for + the v2 brain where several people share one assistant and only + the scoped ``history`` may reach the prompt. Returns: Generated reply text or None @@ -1316,7 +1321,8 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any], # - Plan without it → skip memory work entirely (no keyword LLM, # no diary search, no graph search, no digest LLM). plan_demands_memory = bool(action_plan) and plan_requires_memory(action_plan) - needs_memory = (not action_plan) or plan_demands_memory + # v2 (legacy_memory=False): the scoped history block already answers searchMemory. + needs_memory = legacy_memory and ((not action_plan) or plan_demands_memory) # Recall gate: if the hot-window already carries a fresh tool result # covering the query topic, skip diary/graph enrichment for this turn. @@ -1387,7 +1393,9 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any], dialogue_memory.hot_cache_get(_wp_cache_key) if dialogue_memory and hasattr(dialogue_memory, "hot_cache_get") else None ) - if isinstance(_wp_cached, str): + if not legacy_memory: + debug_log("single-user memory off (v2): no warm profile, diary or graph", "memory") + elif isinstance(_wp_cached, str): warm_profile_block = _wp_cached debug_log("warm profile served from conversation cache", "memory") else: diff --git a/src/jarvis/reply/reply.spec.md b/src/jarvis/reply/reply.spec.md index f3ecd12..068a831 100644 --- a/src/jarvis/reply/reply.spec.md +++ b/src/jarvis/reply/reply.spec.md @@ -62,6 +62,7 @@ Design principles enforced by the engine: - Injected into the system message, after the warm profile and before the diary block, under the label "Information shared in previous conversations" (the slot the model's usual denial names) inside a `` fence. The block says the lines are data, not instructions (they include other people's words), and that Jarvis's own earlier answers are not verified facts. Fence tags inside a line are removed so a record cannot close the fence. - Not gated by the planner: the store read costs no LLM call, and the recent window is needed for follow-ups even when the planner skips memory. `None` or an empty context adds nothing. - Eval: `evals/test_history_context_recall.py` (answers from a matching earlier record; control without the block does not know the fact). + - `legacy_memory=False` (the v2 brain) turns off v1's single-user long-term memory for the turn: no warm profile, and no memory enrichment (no keyword extractor call, diary, graph or memory digest), even when the planner asks for `searchMemory`. v1's profile, diary and graph belong to one user; in a shared v2 channel they would reach other people. The scoped history block stands in for them. 5. Build Initial Messages - messages = [ diff --git a/src/voice_worker/__main__.py b/src/voice_worker/__main__.py index 5d6fa66..9bbd0dd 100644 --- a/src/voice_worker/__main__.py +++ b/src/voice_worker/__main__.py @@ -7,7 +7,7 @@ import asyncio import os import sys -from jarvis.debug import debug_log +from .log import debug_log def main() -> int: diff --git a/src/voice_worker/brain.py b/src/voice_worker/brain.py index 645155b..f6115f6 100644 --- a/src/voice_worker/brain.py +++ b/src/voice_worker/brain.py @@ -3,12 +3,13 @@ interface, so the worker logic is tested with a fake (voice_worker.spec.md, "His from __future__ import annotations +import sys import threading import time from dataclasses import dataclass from typing import Optional, Protocol -from jarvis.debug import debug_log +from .log import debug_log from jarvis.memory.history import HistoryContext, Scope @@ -59,6 +60,44 @@ class MemorySettings: return Scope(chat=self.chat, others=self.others, since=since, off=self.off) +class Muted: + """Stand-in for sys.stdout/sys.stderr that drops every write while a brain reply runs. + + The v1 reply engine and its tools print the question, the answer, tool arguments, URLs and + memory snippets (``print`` and ``debug_log``, also from helper threads such as the web fetch + pool); the worker's logs must never carry such text. Replies run one at a time; the worker's + own log (``voice_worker.log``) writes to the original stderr and is not muted. + """ + + _lock = threading.Lock() + _depth = 0 + + def __init__(self, target): + self._target = target + + def write(self, data): + if Muted._depth: + return len(data) + return self._target.write(data) + + def __getattr__(self, name): + return getattr(self._target, name) + + @classmethod + def run(cls, fn, *args, **kwargs): + with cls._lock: + for name in ("stdout", "stderr"): + stream = getattr(sys, name) + if not isinstance(stream, cls): + setattr(sys, name, cls(stream)) + cls._depth += 1 + try: + return fn(*args, **kwargs) + finally: + with cls._lock: + cls._depth -= 1 + + class JarvisBrain: """The existing jarvis reply engine with the user's history context. @@ -92,8 +131,10 @@ class JarvisBrain: from jarvis.reply.engine import run_reply_engine t0 = time.perf_counter() - answer = run_reply_engine(self.db, self.cfg, None, text, self._memory(user), - language=self.language, history=history) + # legacy_memory=False: v1's profile, diary and graph belong to one user; several people + # share this assistant, so only the asker's scoped history may reach the prompt. + answer = Muted.run(run_reply_engine, self.db, self.cfg, None, text, self._memory(user), + language=self.language, history=history, legacy_memory=False) debug_log(f"brain reply user={user} ok={bool(answer)} ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker") return answer diff --git a/src/voice_worker/log.py b/src/voice_worker/log.py new file mode 100644 index 0000000..03ed144 --- /dev/null +++ b/src/voice_worker/log.py @@ -0,0 +1,17 @@ +"""The worker's own debug log. It writes to the process's original stderr, so it still shows while +the brain's output is muted (``brain.Muted``); otherwise it is ``jarvis.debug.debug_log``.""" + +from __future__ import annotations + +import sys + +from jarvis import debug as _jarvis_debug + + +def debug_log(message: str, category: str = "debug") -> None: + if not _jarvis_debug._is_debug_enabled(): + return + try: + print(f"[{category:^10}] {message}", file=sys.__stderr__, flush=True) + except Exception: + pass diff --git a/src/voice_worker/scheduler.py b/src/voice_worker/scheduler.py index b0ff009..1042f15 100644 --- a/src/voice_worker/scheduler.py +++ b/src/voice_worker/scheduler.py @@ -14,7 +14,7 @@ from typing import Callable, Hashable, Protocol import numpy as np -from jarvis.debug import debug_log +from .log import debug_log @dataclass(frozen=True) diff --git a/src/voice_worker/server.py b/src/voice_worker/server.py index 96d53db..4f6fb4b 100644 --- a/src/voice_worker/server.py +++ b/src/voice_worker/server.py @@ -13,7 +13,7 @@ from typing import Callable import numpy as np -from jarvis.debug import debug_log +from .log import debug_log from .audio import InputConverter, loudness_equalise from .brain import Brain, MemorySettings @@ -57,6 +57,12 @@ class VoiceWorker: self._ids = itertools.count(1) async def start(self) -> None: + if self.history is not None: + # A killed worker never closed its sessions' presence; nobody is heard until the app + # reports who is there again. + n = self.history.close_open_presence(time.time()) + if n: + debug_log(f"closed {n} presence intervals left open by a previous run", "voice_worker") if self.transcriber is not None: self.scheduler = SttScheduler(self.transcriber) self.scheduler.start() @@ -230,12 +236,16 @@ class Session: elif isinstance(ev, Final): debug_log(f"utterance end speaker={speaker} utt={ev.utt} reason={ev.reason} " f"ms={len(ev.audio) * 1000 // SR}", "voice_worker") + # Where, when and by whom it was said, fixed now: decoding can take a while, and a + # move or a late joiner meanwhile must not change who heard it. + said = (self.voice_channel, self.names.get(speaker, speaker), + time.time() - (time.monotonic() - ev.decided_at)) if sched is None: - self._send_final(speaker, ev, SttResult("", 0, "stt unavailable")) + self._send_final(speaker, ev, SttResult("", 0, "stt unavailable"), said) else: sched.submit_final((self.sid, speaker), loudness_equalise(ev.audio, ev.voiced), self._opts(self.settings.final_beam), - lambda res, ev=ev: self._send_final(speaker, ev, res)) + lambda res, ev=ev, said=said: self._send_final(speaker, ev, res, said)) def _partial_cb(self, speaker: str, st: _Speaker, utt: int): def cb(res: SttResult) -> None: @@ -244,10 +254,11 @@ class Session: self.send({"type": "partial", "speaker": speaker, "utt": utt, "text": res.text}) return cb - def _send_final(self, speaker: str, ev: Final, res: SttResult) -> None: + def _send_final(self, speaker: str, ev: Final, res: SttResult, said: tuple) -> None: + channel, name, ts = said store = self.worker.history - if store is not None and self.voice_channel and res.text.strip(): - store.add_voice(self.voice_channel, speaker, self.names.get(speaker, speaker), res.text, time.time()) + if store is not None and channel and res.text.strip(): + store.add_voice(channel, speaker, name, res.text, ts) self.send({"type": "final", "speaker": speaker, "utt": ev.utt, "text": res.text, "end_reason": ev.reason, "audio_ms": len(ev.audio) * 1000 // SR, "wait_ms": int((time.monotonic() - ev.decided_at) * 1000), "stt_ms": res.stt_ms}) diff --git a/src/voice_worker/stt_whisper.py b/src/voice_worker/stt_whisper.py index 5211975..047fc52 100644 --- a/src/voice_worker/stt_whisper.py +++ b/src/voice_worker/stt_whisper.py @@ -10,7 +10,7 @@ import zlib import numpy as np -from jarvis.debug import debug_log +from .log import debug_log from .scheduler import DecodeOptions diff --git a/src/voice_worker/tts.py b/src/voice_worker/tts.py index dadfbc0..1613596 100644 --- a/src/voice_worker/tts.py +++ b/src/voice_worker/tts.py @@ -10,7 +10,7 @@ from typing import Callable, Protocol import numpy as np -from jarvis.debug import debug_log +from .log import debug_log from .sentences import StreamingSplitter diff --git a/src/voice_worker/tts_melo.py b/src/voice_worker/tts_melo.py index ac30ce7..bd506f8 100644 --- a/src/voice_worker/tts_melo.py +++ b/src/voice_worker/tts_melo.py @@ -11,7 +11,7 @@ import threading import numpy as np -from jarvis.debug import debug_log +from .log import debug_log CHILD = os.path.join(os.path.dirname(os.path.abspath(__file__)), "melo_child.py") diff --git a/src/voice_worker/voice_worker.spec.md b/src/voice_worker/voice_worker.spec.md index 6483d49..7a7f1c6 100644 --- a/src/voice_worker/voice_worker.spec.md +++ b/src/voice_worker/voice_worker.spec.md @@ -202,12 +202,16 @@ which carry Discord's own time. <- {"type": "reply_text", "id": "r1", "text": "...", "failed": false} ``` +- At start the worker closes every presence interval a previous run left open (it may have been + killed); the app then reports who is there. - `voice_join` closes the presence of everyone in the previous channel and starts a new one; `voice_leave` and the end of the session close it, because nothing said after that is heard. `member_join` opens the user's presence in the current channel and remembers their name; `member_leave` closes it. Members outside a voice channel are ignored. -- Every non-empty `final` is stored as a `voice` record of the current channel under the speaker's - name (the user ID until `member_join` names them). A final outside a voice channel is not stored. +- Every non-empty `final` is stored as a `voice` record under the channel, the speaker's name (the + user ID until `member_join` names them) and the time of the moment its end was decided, all fixed + then: decoding takes a while, and a move, a rename or a late joiner meanwhile must not change where + or when it was said, or who heard it. A final said outside a voice channel is not stored. - Chat messages are stored only as given; the app sends only registered channels. - `reply` asks the brain to answer `user`'s `text`. The worker builds the history context for that user (their scope only) and runs the reply engine on a worker thread, one reply at a time. @@ -216,7 +220,12 @@ which carry Discord's own time. answer is stored when the app reports the sent message as `chat_message` with `to`. With `speak: true` the answer is also spoken through the TTS path under the same id (audio frames and `tts_done`), and `tts_cancel` with that id stops it. Each user keeps their own short-term dialogue - memory (tool carry-over) across replies. + memory (tool carry-over) across replies. The brain runs the reply engine with v1's single-user + memory off (`legacy_memory=False`: no warm profile, diary or graph), because several people share + this assistant and only the asker's scoped history may reach the prompt. +- While a reply runs, the process's stdout and stderr are muted: the v1 engine and its tools print + questions, answers, tool arguments and memory text, also from helper threads. The worker's own log + (`voice_worker/log.py`) writes to the original stderr and keeps working. - Memory settings, optional in `hello` as `"memory": {...}` and changeable with `{"type": "memory_config", ...}`: `chat` (true), `others` (true), `off` (false), `since_days` (null: whole history), `recent_minutes` (30), `recent_records` (20), `matches` (5), diff --git a/tests/test_engine_history_context.py b/tests/test_engine_history_context.py index 3dabe4c..4bb0433 100644 --- a/tests/test_engine_history_context.py +++ b/tests/test_engine_history_context.py @@ -49,3 +49,26 @@ def test_history_lines_reach_the_system_prompt(): @pytest.mark.parametrize("history", [None, HistoryContext()]) def test_no_history_adds_nothing(history): assert "" not in _system_prompt(history) + + +def _system_prompt_with(legacy_memory, profile): + with patch("src.jarvis.reply.engine.plan_query", return_value=[]), \ + patch("src.jarvis.reply.engine.extract_search_params_for_memory", return_value={}), \ + patch("src.jarvis.reply.engine.extract_text_from_response", return_value="Hello."), \ + patch("src.jarvis.memory.graph.GraphMemoryStore"), \ + patch("src.jarvis.memory.graph_ops.build_warm_profile", return_value={"user": profile, "directives": ""}), \ + patch("src.jarvis.memory.graph_ops.format_warm_profile_block", return_value=profile), \ + patch("src.jarvis.reply.engine.chat_with_messages") as mock_chat: + mock_chat.return_value = {"message": {"content": "Hello."}} + run_reply_engine(db=Mock(), cfg=_mock_cfg(512), tts=None, text="hi", dialogue_memory=DialogueMemory(), + history=HistoryContext(), legacy_memory=legacy_memory) + messages = mock_chat.call_args.kwargs.get("messages") or mock_chat.call_args.args[2] + return messages[0]["content"] + + +@pytest.mark.unit +def test_v2_turns_off_the_single_user_profile(): + """v1's profile, diary and graph belong to one user; in a shared v2 channel they would leak.""" + profile = "PROFILE: Alice is allergic to peanuts" + assert profile in _system_prompt_with(True, profile) + assert profile not in _system_prompt_with(False, profile) diff --git a/tests/test_voice_worker_brain.py b/tests/test_voice_worker_brain.py index c24ea1c..029e5ba 100644 --- a/tests/test_voice_worker_brain.py +++ b/tests/test_voice_worker_brain.py @@ -225,3 +225,122 @@ def test_malformed_history_messages_are_rejected(store): seen = asyncio.run(with_worker(body, store=store, brain=FakeBrain())) assert [s[-1]["code"] for s in seen] == ["bad_message", "bad_message"] + + +class SlowTranscriber(LevelTranscriber): + def __init__(self, delay): + super().__init__() + self.delay = delay + + def transcribe(self, clips, opts): + import time as _t + _t.sleep(self.delay) + return super().transcribe(clips, opts) + + +@pytest.mark.unit +def test_a_slow_final_keeps_the_channel_time_and_name_of_when_it_was_said(store): + async def body(url): + async with connect(url) as ws: + await hello(ws, stt={"partials": False}) + await send(ws, type="voice_join", channel="vc1") + await send(ws, type="member_join", user="100", name="Alice") + await stream(ws, "100", 0.9, 600) + await send(ws, type="speaking_end", speaker="100") + # While the model is still decoding: a late joiner arrives, then Jarvis moves. + await asyncio.sleep(0.15) + await send(ws, type="member_join", user="300", name="Late") + await send(ws, type="member_join", user="100", name="Renamed") + await send(ws, type="voice_join", channel="vc2") + await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "final") + + worker = VoiceWorker(TOKEN, SlowTranscriber(0.6), AmplitudeVad, None, history=store) + + async def run(): + server = await worker.serve("127.0.0.1", 0) + port = server.sockets[0].getsockname()[1] + try: + await asyncio.wait_for(body(f"ws://127.0.0.1:{port}"), 15) + finally: + server.close() + await server.wait_closed() + await worker.stop() + + asyncio.run(run()) + recs = store.records_of("100") + assert [(r.channel_id, r.author_name, r.text) for r in recs] == [("vc1", "Alice", "level 0.9")] + # The late joiner was not there when it was said. + assert store.recent("300", now=4_000_000_000, scope=Scope(), minutes=10 ** 9) == [] + + +@pytest.mark.unit +def test_presence_left_open_by_a_previous_run_is_closed_at_start(store): + store.presence_join("old", "vc1", 1_000.0) # the worker was killed with "old" still present + + async def body(url): + async with connect(url) as ws: + await hello(ws, stt={"partials": False}) + await send(ws, type="voice_join", channel="vc1") + await send(ws, type="member_join", user="100", name="Alice") + await say(ws, "100", 0.9) + + asyncio.run(with_worker(body, store=store)) + assert texts(store.recent("100", now=4_000_000_000, scope=Scope(), minutes=10 ** 9)) == ["level 0.9"] + assert store.recent("old", now=4_000_000_000, scope=Scope(), minutes=10 ** 9) == [] + + +@pytest.mark.unit +def test_the_brain_prints_no_text_from_its_thread(tmp_path, monkeypatch, capfd): + pytest.importorskip("mcp") # the real brain needs the jarvis dependencies (--brain) + import sys + import threading + + from jarvis.memory.history import HistoryContext + from voice_worker import brain as brain_mod + + cfg = tmp_path / "config.json" + cfg.write_text(json.dumps({"db_path": str(tmp_path / "jarvis.db")})) + monkeypatch.setenv("JARVIS_CONFIG_PATH", str(cfg)) + + def fake_engine(db, cfg, tts, text, mem, language=None, history=None, legacy_memory=True): + print(f"answer text {text}") + print(f"debug {text}", file=sys.stderr) + # Tools print from helper threads too (the web fetch pool). + helper = threading.Thread(target=lambda: print(f"helper {text}", file=sys.stderr)) + helper.start() + helper.join() + return f"answer to {text}" + + import jarvis.reply.engine as engine + monkeypatch.setattr(engine, "run_reply_engine", fake_engine) + b = brain_mod.JarvisBrain() + out = [] + t = threading.Thread(target=lambda: out.append(b.reply("100", "secret question", HistoryContext()))) + t.start() + t.join() + print("other output stays") + captured = capfd.readouterr() + assert out == ["answer to secret question"] + assert "secret question" not in captured.out + captured.err + assert "other output stays" in captured.out + + +@pytest.mark.unit +def test_the_brain_turns_off_the_single_user_memory(tmp_path, monkeypatch): + pytest.importorskip("mcp") # the real brain needs the jarvis dependencies (--brain) + from jarvis.memory.history import HistoryContext + from voice_worker import brain as brain_mod + + cfg = tmp_path / "config.json" + cfg.write_text(json.dumps({"db_path": str(tmp_path / "jarvis.db")})) + monkeypatch.setenv("JARVIS_CONFIG_PATH", str(cfg)) + seen = {} + + def fake_engine(db, cfg, tts, text, mem, language=None, history=None, legacy_memory=True): + seen["legacy_memory"] = legacy_memory + return "ok" + + import jarvis.reply.engine as engine + monkeypatch.setattr(engine, "run_reply_engine", fake_engine) + brain_mod.JarvisBrain().reply("100", "q", HistoryContext()) + assert seen["legacy_memory"] is False