From 6b23a8adc74772b96bd1293a92f70fefb8a2fda1 Mon Sep 17 00:00:00 2001 From: javis-bot Date: Wed, 11 Nov 2026 12:32:48 +0900 Subject: [PATCH] feat(llm): talk to OpenAI-compatible servers such as Strata A base URL that ends in /v1 now selects the OpenAI-compatible API: chat goes to /chat/completions and embeddings to /embeddings. The conversion lives in src/jarvis/llm.py only, so the reply engine and every other caller keep their Ollama-shaped messages and responses. - Requests drop private engine keys, send tool-call arguments as JSON strings, map num_predict to max_tokens and the thinking switch to chat_template_kwargs.enable_thinking, and leave out num_ctx and keep_alive. - Responses come back with message.content, tool calls with dict arguments and an id, and message.thinking from reasoning_content. - Streaming, the warm-up (one-token chat completion), the intent judge and the dictation clean-up use the same path. A server without an embedding model fails open to the text paths. - Specs and docs/llm_contexts.md describe the transport; roadmap records M3 step 1. Co-Authored-By: Claude Opus 5.5 --- docs/llm_contexts.md | 2 + docs/v2/roadmap.md | 2 +- src/jarvis/dictation/dictation.spec.md | 3 +- src/jarvis/dictation/dictation_engine.py | 9 + src/jarvis/listening/intent_judge.py | 77 +++++--- src/jarvis/listening/listening.spec.md | 2 +- src/jarvis/llm.py | 231 ++++++++++++++++++++++ src/jarvis/memory/embeddings.py | 24 +++ src/jarvis/reply/reply.spec.md | 4 +- src/jarvis/tools/selection.spec.md | 2 +- tests/test_llm_openai_compat.py | 233 +++++++++++++++++++++++ 11 files changed, 558 insertions(+), 31 deletions(-) create mode 100644 tests/test_llm_openai_compat.py diff --git a/docs/llm_contexts.md b/docs/llm_contexts.md index 37e63a5..0b67724 100644 --- a/docs/llm_contexts.md +++ b/docs/llm_contexts.md @@ -2,6 +2,8 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it is gated. This is the reference for optimising the app's main bottleneck (LLM latency). Keep it in sync with the code — see the note at the bottom. +**Transport (all contexts below).** `ollama_base_url` selects the API. A URL that ends in `/v1` (v2: Strata at `http://127.0.0.1:8080/v1`; also llama.cpp or Ollama's own `/v1`) uses the OpenAI-compatible API: chat goes to `/chat/completions`, embeddings to `/embeddings`, and the warm-up is a one-token chat completion. Any other URL uses Ollama's `/api/chat`, `/api/generate` and `/api/embeddings`. The conversion lives in `src/jarvis/llm.py` only: callers keep sending Ollama-shaped messages and reading Ollama-shaped responses (`message.content`, `message.tool_calls` with dict arguments, `message.thinking` from `reasoning_content`). On the OpenAI path `num_ctx` and `keep_alive` are not sent (the server owns its context size and residency), `num_predict` becomes `max_tokens`, and the thinking switch becomes `chat_template_kwargs.enable_thinking`. The `num_ctx: 8192` limits listed below apply to the Ollama path. A server with no embedding model (Strata) answers `/embeddings` with an error and every embedding consumer fails open to its text path. + --- ## 1. Main Reply Loop (agentic messages loop) diff --git a/docs/v2/roadmap.md b/docs/v2/roadmap.md index 3a220c6..a3f831f 100644 --- a/docs/v2/roadmap.md +++ b/docs/v2/roadmap.md @@ -7,7 +7,7 @@ | M0 | 게임PC 사전 검증 | 측정 스크립트를 게임PC에서 직접 실행하고 결과 파일을 보내 주시는 방식입니다. 결과로 기본값을 확정합니다. 확인 항목은 아래 목록에 있습니다. 가상 모니터 확인을 뺀 모든 항목 측정 완료(2026-11-10). 가상 모니터 확인은 M6에서 합니다. 2026-11-10에 추가한 `desktop` 단계(agent-desktop)는 측정을 마쳤고, 자동 성능 조절용 `gamemode` 단계는 세 번 쟀고, `gamemode_stt`(LLM과 STT를 함께 켬, STT GPU·CPU)와 `gamemode_live`(실제 게임을 켬) 단계도 측정을 마쳤습니다(2026-11-10). 결과로 게임 중 최소 배치는 전문가 2GB로 정했고, 평소 배치에 STT 자리를 남기는 것과 게임 중 LLM이 답하는 동안 부분 전사 간격을 늘리는 것(게임이 없을 때는 그대로)도 정했습니다(사용자 결정, 2026-11-10). `gamemode_tune` 단계도 측정을 마쳤고(2026-11-11), 게임 중 LLM이 답하는 동안에는 부분 전사를 멈추기로 정했습니다. GPU 95% 제한은 측정에서 사용률을 낮추지 못해 기본값을 끔으로 바꿨고(사용자 결정, 2026-11-11), `video_drops` 단계(Jarvis를 돌리는 동안 유튜브 영상의 끊긴 프레임, 직접 실행)도 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 영상이 끊기지 않았고 제한이 끊김을 줄이지 못해 설정을 뺐습니다. 대신 게임 화면의 프레임을 직접 재는 `game_frames` 단계(게임을 켠 채로 Jarvis를 끈 때·쉴 때·바쁠 때와 우선순위를 낮춘 양보 상태, 직접 실행, 관리자 권한)를 넣었고 측정을 마쳤습니다(2026-11-11). Jarvis가 바쁠 때도 게임의 평균 fps와 1% low가 5% 넘게 떨어지지 않아 양보는 넣지 않기로 했습니다. 남은 M0 측정은 가상 모니터 확인(M6)입니다. 측정 키트: [scripts/m0/](../../scripts/m0/README.md) | | M1 | 음성 워커 | 화자별 스트리밍 STT(공용 모델, 배치), 문장 단위 TTS를 WebSocket으로 제공. 녹음 파일 10개 동시 입력 테스트 통과. 구현과 테스트 완료(2026-11-11): `src/voice_worker/`, 스펙은 [voice_worker.spec.md](../../src/voice_worker/voice_worker.spec.md). 이 서버의 RTX 5050에서 10명이 겹쳐 말하는 실시간 입력(48kHz 스테레오, M0 한국어 샘플 8개를 10명에게 나눠 줌, 절반은 디스코드 speaking 종료, 절반은 무음으로 끝남)을 넣어 10명 모두 최종 전사를 받았고 문자 오류율은 평균 4%, 최대 19%("일곱 시"를 "7시"로 적은 것), 말 끝 판정부터 최종 전사 전송까지 중앙값 0.64초, 최대 1.56초였습니다. TTS는 첫 문장 소리가 0.2초에 나왔고, 그 소리를 다시 넣으면 같은 문장으로 받아 적혔습니다. 샘플이 MeloTTS 합성 음성이라 실제 사람 녹음과 게임PC 수치는 아직 없습니다 | | M2 | 디스코드 음성 코어 이전 | v1 봇 기능을 새 앱 안으로 이전. 입장/퇴장, 화자별 수신, 재생, 정지 명령 동작. 채팅 채널 연결(읽기·보내기). 구현과 테스트 완료(2026-11-11): `app/`(Rust, serenity + songbird), 스펙은 [discord_core.spec.md](../../app/src/discord_core.spec.md). 처음 Electron + TypeScript로 만든 것을 사용자 요청으로 Rust로 다시 씀. 앱이 음성 워커를 띄우고(실행마다 새 토큰, 죽으면 다시 시작) 연결이 끊기면 다시 붙습니다. 음성 채널에 들어갈 때마다 전사 고지를 채널 채팅에 올립니다. songbird 0.6.1의 DAVE 수신 해독이 조각 3분의 1가량을 깨뜨려서(20ms마다 음이 바뀌는 시험 신호로 확인, 보내기는 정상) 받은 패킷의 순서 맞추기와 Opus 해독은 앱이 직접 합니다. 실제 디스코드에서 봇 두 개로 확인(이 서버의 RTX 5050): 다른 봇이 채널에 튼 녹음 문장 5개를 모두 받아 적었고(문자 오류율 3개 0%, "기록해"를 "비록해"로 적은 1개 5%, "일곱 시"를 "7시"로 적은 1개 12.5%), 말이 끝난 뒤 최종 전사까지 0.71~0.90초였습니다. 자비스가 말한 문장을 상대 봇이 받아 다시 전사하니 오류율 3.4%였고, 정지 명령 뒤 0.08초 만에 소리가 끊겼습니다. 채팅 채널은 새 글 받기, 최근 글 읽기, 보내기(2,000자 넘으면 나눔, 아무도 호출하지 않음)가 됩니다. 사람이 직접 말한 소리와 게임PC(Windows) 빌드는 아직 확인하지 않았고, 음성으로 "자비스, 멈춰"는 호출 판정이 필요해 M4에서 합니다 | -| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과 | +| M3 | 두뇌 연결 | 기존 jarvis를 Ollama 대신 Strata 주소로 연결. 대화 기록 저장소와 지난 대화 전체 검색(본인 기록 + 등록한 채팅 채널 + 같은 음성 채널의 다른 사람 말). 채팅 채널 도구(읽기·검색·보내기·수정·반응·삭제·고정). 메모리, 날씨, 웹 검색, 식단, MCP 기존 테스트와 평가(evals) 통과. 1단계 완료(2026-11-11): LLM 주소가 `/v1`로 끝나면 OpenAI 호환 방식(`/chat/completions`, `/embeddings`)으로 부름. 형식 변환은 `src/jarvis/llm.py` 한곳에서 하고 두뇌의 나머지 코드는 그대로 씀. 대화, 도구 호출, 스트리밍, 의도 판정, 받아쓰기 정리, 미리 불러오기가 모두 이 방식으로 감. 이 서버 Ollama의 `/v1` 주소와 qwen2.5:7b로 실제 두뇌 한 턴을 돌려 "서울 날씨 어때?"에 날씨 도구를 부르고 그 결과로 답하는 것을 확인함. Strata에는 임베딩 모델이 없어 지금은 지난 기록 검색이 글자 검색으로만 돌아감(2단계에서 정함) | | M4 | 왕복 완성과 지연 튜닝 | 호출어·호출 방식·지정한 사람 설정, 동시 대화 응답 정책 4가지(기본 "처음 대화만")와 정책별 끼어들기 규칙. 말 끝부터 첫 소리까지 실측 1.5초 이내, 10명 채널 부하 테스트 | | M5 | 관리 앱 기본 | 첫 실행 마법사, 데이터 폴더, 런타임·모델 자동 설치, 대시보드, 봇/채널·호출/채팅 채널(디스코드 채널 이름·주제 반영 포함)/대화 기억/모델 설정, 로그 화면(보기·복사·다운로드) | | M6 | 기능 화면과 브라우저 | 기능 켜기/끄기, MCP 관리, 봇용 크롬(가상 모니터에 둠, Virtual Display Driver 포함), 예약 수집 틀(대상은 비움). 디스코드 "화면" 탭 공유, 커서, 모니터 해제 때 창 위치 확인 | diff --git a/src/jarvis/dictation/dictation.spec.md b/src/jarvis/dictation/dictation.spec.md index 12b2bb9..986767d 100644 --- a/src/jarvis/dictation/dictation.spec.md +++ b/src/jarvis/dictation/dictation.spec.md @@ -57,7 +57,8 @@ After transcription, text passes through these stages in order: 1. **Custom dictionary** — case-insensitive whole-word regex replacements from `dictation_custom_dictionary`. Each entry is `"wrong -> right"`. 2. **LLM filler removal** (optional) — when `dictation_filler_removal` is - enabled, sends the text to the local Ollama instance (same model as the + enabled, sends the text to the local LLM server (Ollama, or the + OpenAI-compatible `/chat/completions` when the URL ends in `/v1`; same model as the assistant) with a prompt to remove filler words (um, uh, like, you know, etc.) while preserving meaning. Uses a 5-second timeout; falls back to the unprocessed text on failure. diff --git a/src/jarvis/dictation/dictation_engine.py b/src/jarvis/dictation/dictation_engine.py index d87e6fa..113777f 100644 --- a/src/jarvis/dictation/dictation_engine.py +++ b/src/jarvis/dictation/dictation_engine.py @@ -450,6 +450,15 @@ def _llm_clean_dictation(text: str, ollama_base_url: str, model: str = "gemma4:e f"{text}" ) + from ..llm import call_llm_direct, is_openai_compatible + + if is_openai_compatible(ollama_base_url): + cleaned = (call_llm_direct(ollama_base_url, model, "", prompt, timeout_sec=5, thinking=thinking) or "").strip() + if cleaned: + debug_log(f"LLM filler removal: {text!r} → {cleaned!r}", "dictation") + return cleaned + return text + try: resp = requests.post( f"{ollama_base_url}/api/generate", diff --git a/src/jarvis/listening/intent_judge.py b/src/jarvis/listening/intent_judge.py index 631efc3..bebf782 100644 --- a/src/jarvis/listening/intent_judge.py +++ b/src/jarvis/listening/intent_judge.py @@ -13,6 +13,7 @@ from dataclasses import dataclass from typing import Optional, List from ..debug import debug_log +from ..llm import openai_chat_body, extract_text_from_response, is_openai_compatible, post_chat_completion from .transcript_buffer import TranscriptSegment try: @@ -32,6 +33,18 @@ def warm_up_ollama_model(base_url: str, model: str, timeout: float) -> bool: """ if not REQUESTS_AVAILABLE or not base_url or not model: return False + if is_openai_compatible(base_url): + # The server loads its model at launch; one token checks it answers. + body = openai_chat_body(model, [{"role": "user", "content": "hi"}], stream=False, + thinking=False, max_tokens=1) + try: + response = post_chat_completion(base_url, body, timeout) + ok = response.status_code == 200 + debug_log(f"llm warmup {'ok' if ok else f'failed HTTP {response.status_code}'} (model={model})", "voice") + return ok + except Exception as e: + debug_log(f"llm warmup error (model={model}): {e}", "voice") + return False try: response = requests.post( f"{base_url}/api/generate", @@ -418,42 +431,56 @@ Examples: # (which was the original timeout culprit). Ollama's default is # 5m; we pin to 30m because voice sessions can have long quiet # stretches and reloading mid-conversation is a bad experience. - response = requests.post( - f"{self.config.ollama_base_url}/api/generate", - json={ - "model": self.config.model, - "prompt": user_prompt, - "system": system_prompt, - "stream": False, - "think": self.config.thinking, - "keep_alive": "30m", - "options": { - "temperature": 0.0, - "num_predict": 200, - # Headroom for: ~2k-token system prompt + up to 2 minutes - # of chatty multi-speaker transcript (default - # transcript_buffer_duration_sec=120 in listener.py). - # 4096 was cutting close to 90% utilisation in the - # worst case after the prompt grew in PR #362, which - # risks silent ollama truncation of the system - # prompt's tail. - "num_ctx": 8192, + if is_openai_compatible(self.config.ollama_base_url): + response = post_chat_completion( + self.config.ollama_base_url, + openai_chat_body( + self.config.model, + [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}], + stream=False, thinking=self.config.thinking, max_tokens=200, temperature=0.0, + ), + self.config.timeout_sec, + ) + else: + response = requests.post( + f"{self.config.ollama_base_url}/api/generate", + json={ + "model": self.config.model, + "prompt": user_prompt, + "system": system_prompt, + "stream": False, + "think": self.config.thinking, + "keep_alive": "30m", + "options": { + "temperature": 0.0, + "num_predict": 200, + # Headroom for: ~2k-token system prompt + up to 2 minutes + # of chatty multi-speaker transcript (default + # transcript_buffer_duration_sec=120 in listener.py). + # 4096 was cutting close to 90% utilisation in the + # worst case after the prompt grew in PR #362, which + # risks silent ollama truncation of the system + # prompt's tail. + "num_ctx": 8192, + }, }, - }, - timeout=self.config.timeout_sec, - ) + timeout=self.config.timeout_sec, + ) if response.status_code != 200: # Don't back off on transient HTTP errors — voice is high-turn # and a 503 from an overloaded Ollama shouldn't kill the next # 30s of intent judging. Retry on the next engagement signal. - reason = f"HTTP {response.status_code} from Ollama" + reason = f"HTTP {response.status_code} from the LLM server" debug_log(f"intent judge: {reason}", "voice") self._last_failure_reason = reason return None result = response.json() - response_text = result.get("response", "") + if is_openai_compatible(self.config.ollama_base_url): + response_text = extract_text_from_response(result) or "" + else: + response_text = result.get("response", "") judgment = self._parse_response(response_text) diff --git a/src/jarvis/listening/listening.spec.md b/src/jarvis/listening/listening.spec.md index de5e942..97864c5 100644 --- a/src/jarvis/listening/listening.spec.md +++ b/src/jarvis/listening/listening.spec.md @@ -114,7 +114,7 @@ On small models, a caveat line is appended above a more involved example to set **What gets warmed:** - **Whisper** — loading the model; additionally a silent-audio transcribe so the first real utterance doesn't pay the cold-decode cost. Both the MLX and faster-whisper backends do this. -- **Chat model** (`cfg.ollama_chat_model`) — a minimal Ollama `/api/generate` request with `keep_alive=30m` so the weights stay resident. +- **Chat model** (`cfg.ollama_chat_model`) — a minimal Ollama `/api/generate` request with `keep_alive=30m` so the weights stay resident. When `ollama_base_url` ends in `/v1` (an OpenAI-compatible server such as Strata, which loads its model at launch) the warm-up is a one-token `/chat/completions` request that only checks the server answers, and the intent judge itself also calls `/chat/completions`. - **Intent judge model** (`cfg.intent_judge_model`) — same pattern. If it points at the same Ollama model as the chat model, a single warmup covers both roles (Ollama loads the weights once). **Concurrency:** LLM warmups run in daemon threads started before Whisper loads, so they overlap with Whisper initialisation. After Whisper finishes, the listener joins the warmup threads with a **single 60 s budget** shared across them all. If the budget is exhausted, the listener continues (with a `⏳ Some models still warming — continuing anyway` notice) and the first engagement pays the cold-load cost on demand. diff --git a/src/jarvis/llm.py b/src/jarvis/llm.py index c34f3bb..fa0cf65 100644 --- a/src/jarvis/llm.py +++ b/src/jarvis/llm.py @@ -6,6 +6,8 @@ import os import sys import requests import json +import time +import uuid def _caller_name() -> str: @@ -35,6 +37,152 @@ class ToolsNotSupportedError(Exception): pass +# --------------------------------------------------------------------------- +# OpenAI-compatible servers (Strata, llama.cpp, Ollama's own /v1) +# --------------------------------------------------------------------------- +# +# A base URL that ends in ``/v1`` selects the OpenAI chat-completions API. +# Callers keep building Ollama-shaped messages and reading Ollama-shaped +# responses (``message.content``, ``message.tool_calls`` with dict arguments, +# ``message.thinking``); the conversion lives here only. The server owns the +# context size, so ``num_ctx`` and ``keep_alive`` are not sent. + +_OPENAI_MESSAGE_KEYS = ("role", "content", "tool_calls", "tool_call_id", "name") + + +def is_openai_compatible(base_url: str) -> bool: + """True when ``base_url`` points at an OpenAI-compatible API (ends in ``/v1``).""" + return bool(base_url) and base_url.rstrip("/").endswith("/v1") + + +def _openai_messages(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Copy Ollama-shaped messages into OpenAI ones without changing the originals. + + Private engine keys (``_is_tool_guidance``), ``thinking``, ``tool_name`` and + ``tool_failed`` are dropped, and tool-call arguments become JSON strings. + """ + out: List[Dict[str, Any]] = [] + for m in messages: + msg = {k: m[k] for k in _OPENAI_MESSAGE_KEYS if k in m} + if msg.get("content") is None: + msg["content"] = "" + calls = m.get("tool_calls") + if calls: + converted = [] + for i, tc in enumerate(calls): + fn = tc.get("function") or {} + args = fn.get("arguments") + if not isinstance(args, str): + args = json.dumps(args or {}, ensure_ascii=False) + converted.append({ + "id": tc.get("id") or f"call_{i}", + "type": "function", + "function": {"name": fn.get("name", ""), "arguments": args}, + }) + msg["tool_calls"] = converted + else: + msg.pop("tool_calls", None) + out.append(msg) + return out + + +def openai_chat_body( + chat_model: str, + messages: List[Dict[str, Any]], + *, + stream: bool, + thinking: bool, + max_tokens: Optional[int] = None, + temperature: Optional[float] = None, + tools: Optional[List[Dict[str, Any]]] = None, +) -> Dict[str, Any]: + body: Dict[str, Any] = { + "model": chat_model, + "messages": _openai_messages(messages), + "stream": stream, + # Qwen-family templates think unless told not to; spoken answers must not. + "chat_template_kwargs": {"enable_thinking": bool(thinking)}, + } + if max_tokens is not None and max_tokens > 0: + body["max_tokens"] = int(max_tokens) + if temperature is not None: + body["temperature"] = temperature + if tools: + body["tools"] = tools + return body + + +def post_chat_completion(base_url: str, body: Dict[str, Any], timeout_sec: float, stream: bool = False): + """POST ``body`` to ``/chat/completions`` and return the response.""" + return requests.post( + f"{base_url.rstrip('/')}/chat/completions", json=body, timeout=timeout_sec, stream=stream + ) + + +def _parse_tool_arguments(raw: Any) -> Dict[str, Any]: + if isinstance(raw, dict): + return raw + if isinstance(raw, str) and raw.strip(): + try: + parsed = json.loads(raw) + if isinstance(parsed, dict): + return parsed + except json.JSONDecodeError: + pass + debug_log(f"tool-call arguments are not a JSON object (len={len(raw)})", "llm") + return {} + + +def _ollama_shaped(data: Dict[str, Any]) -> Optional[Dict[str, Any]]: + """Turn an OpenAI chat completion into the Ollama response shape callers read.""" + choices = data.get("choices") if isinstance(data, dict) else None + if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict): + return None + choice = choices[0] + src = choice.get("message") or {} + msg: Dict[str, Any] = {"role": "assistant", "content": src.get("content") or ""} + reasoning = src.get("reasoning_content") or src.get("reasoning") + if reasoning: + msg["thinking"] = reasoning + calls = [] + for tc in src.get("tool_calls") or []: + fn = tc.get("function") or {} + calls.append({ + "id": tc.get("id") or f"call_{uuid.uuid4().hex[:8]}", + "type": "function", + "function": {"name": fn.get("name", ""), "arguments": _parse_tool_arguments(fn.get("arguments"))}, + }) + if calls: + msg["tool_calls"] = calls + return {"message": msg, "done": True, "done_reason": choice.get("finish_reason"), "usage": data.get("usage")} + + +def _log_openai_timing(data: Dict[str, Any], elapsed: float, caller: str) -> None: + """Per-call latency line for OpenAI-compatible servers (no server-side phase split).""" + try: + usage = data.get("usage") or {} + print( + f" ⏱️ llm[{caller}] total={elapsed:.2f}s " + f"prompt={usage.get('prompt_tokens')}t gen={usage.get('completion_tokens')}t", + flush=True, + ) + except Exception: # pragma: no cover - logging must never break a reply + pass + + +def _openai_chat(base_url: str, body: Dict[str, Any], timeout_sec: float, caller: str) -> Dict[str, Any]: + """Non-streaming chat completion, returned in the Ollama shape. Raises requests errors.""" + t0 = time.perf_counter() + with post_chat_completion(base_url, body, timeout_sec) as resp: + resp.raise_for_status() + data = resp.json() + _log_openai_timing(data, time.perf_counter() - t0, caller) + shaped = _ollama_shaped(data) + if shaped is None: + raise ValueError(f"no choices in response keys={list(data.keys()) if isinstance(data, dict) else type(data)}") + return shaped + + def _log_ollama_timing(data: Dict[str, Any], num_ctx: int, caller: str) -> None: """Emit a one-line per-call latency breakdown so a slow voice turn can be attributed to a specific internal LLM call (router / enrichment / digest / @@ -81,6 +229,20 @@ def call_llm_direct(base_url: str, chat_model: str, system_prompt: str, user_con {"role": "user", "content": user_content} ] + if is_openai_compatible(base_url): + caller = _caller_name() + body = openai_chat_body(chat_model, messages, stream=False, thinking=thinking, temperature=temperature) + try: + content = _openai_chat(base_url, body, timeout_sec, caller)["message"]["content"] + if content.strip(): + return content + debug_log("call_llm_direct: empty content from OpenAI-compatible server", "llm") + except requests.exceptions.Timeout: + debug_log(f"call_llm_direct: timeout after {timeout_sec}s", "llm") + except Exception as e: + debug_log(f"call_llm_direct: request failed — {e}", "llm") + return None + options: Dict[str, Any] = {"num_ctx": num_ctx} if temperature is not None: options["temperature"] = temperature @@ -150,6 +312,9 @@ def call_llm_streaming( {"role": "user", "content": user_content} ] + if is_openai_compatible(base_url): + return _openai_stream(base_url, chat_model, messages, on_token, timeout_sec, thinking) + payload: Dict[str, Any] = { "model": chat_model, "messages": messages, @@ -197,6 +362,49 @@ def call_llm_streaming( return None +def _openai_stream( + base_url: str, + chat_model: str, + messages: List[Dict[str, Any]], + on_token: Optional[Callable[[str], None]], + timeout_sec: float, + thinking: bool, +) -> Optional[str]: + """Server-sent-events stream from ``/chat/completions``; answer text only (no reasoning).""" + body = openai_chat_body(chat_model, messages, stream=True, thinking=thinking) + try: + with post_chat_completion(base_url, body, timeout_sec, stream=True) as resp: + resp.raise_for_status() + full_response = [] + for line in resp.iter_lines(): + if not line: + continue + text = line.decode("utf-8", "replace") if isinstance(line, bytes) else line + if not text.startswith("data:"): + continue + payload = text[5:].strip() + if payload == "[DONE]": + break + try: + chunk = json.loads(payload) + except json.JSONDecodeError: + continue + for choice in chunk.get("choices") or []: + content = (choice.get("delta") or {}).get("content") + if content: + full_response.append(content) + if on_token: + on_token(content) + result = "".join(full_response) + return result if result.strip() else None + except requests.exceptions.Timeout: + debug_log(f"call_llm_streaming: timeout after {timeout_sec}s", "llm") + return None + except Exception as e: + debug_log(f"call_llm_streaming: request failed — {e}", "llm") + return None + + def extract_text_from_response(data: Dict[str, Any]) -> Optional[str]: """Extract text from LLM response - supports multiple response formats.""" # Preferred: Ollama chat non-stream format @@ -251,6 +459,29 @@ def chat_with_messages( Returns the parsed JSON response dict on success, or None on error/timeout. """ + if is_openai_compatible(base_url): + opts = extra_options if isinstance(extra_options, dict) else {} + body = openai_chat_body( + chat_model, messages, stream=False, thinking=thinking, + max_tokens=opts.get("num_predict"), temperature=opts.get("temperature"), tools=tools, + ) + caller = _caller_name() + try: + return _openai_chat(base_url, body, timeout_sec, caller) + except requests.exceptions.Timeout: + print(" ⏱️ LLM request timed out", flush=True) + except requests.exceptions.ConnectionError as e: + print(f" ❌ LLM connection error: {e}", flush=True) + except requests.exceptions.HTTPError as e: + if e.response is not None and e.response.status_code == 400 and tools: + raise ToolsNotSupportedError( + f"Model {chat_model!r} returned HTTP 400 — native tools API not supported" + ) + print(f" ❌ LLM HTTP error: {e}", flush=True) + except Exception as e: + print(f" ❌ LLM error: {e}", flush=True) + return None + # Main agentic chat uses 8192 so the system prompt (tool list + protocol # guidance + memory context) doesn't overflow and force ollama to truncate # — which previously dropped the tool schema on smaller models like diff --git a/src/jarvis/memory/embeddings.py b/src/jarvis/memory/embeddings.py index 49d3999..3125863 100644 --- a/src/jarvis/memory/embeddings.py +++ b/src/jarvis/memory/embeddings.py @@ -2,7 +2,31 @@ from __future__ import annotations import requests +def _openai_embedding(text: str, base_url: str, model: str, timeout_sec: float) -> list[float] | None: + """``/embeddings`` of an OpenAI-compatible server. A server without an + embedding model (Strata serves the chat model only) answers an error, and + the caller falls back to text search.""" + try: + resp = requests.post( + f"{base_url.rstrip('/')}/embeddings", + json={"model": model, "input": text}, + timeout=timeout_sec, + ) + resp.raise_for_status() + data = resp.json().get("data") or [] + vec = data[0].get("embedding") if data and isinstance(data[0], dict) else None + if isinstance(vec, list): + return [float(x) for x in vec] + except Exception: + return None + return None + + def get_embedding(text: str, base_url: str, model: str, timeout_sec: float = 15.0) -> list[float] | None: + from ..llm import is_openai_compatible + + if is_openai_compatible(base_url): + return _openai_embedding(text, base_url, model, timeout_sec) try: resp = requests.post( f"{base_url.rstrip('/')}/api/embeddings", diff --git a/src/jarvis/reply/reply.spec.md b/src/jarvis/reply/reply.spec.md index e3c5f26..fcb10ab 100644 --- a/src/jarvis/reply/reply.spec.md +++ b/src/jarvis/reply/reply.spec.md @@ -77,7 +77,7 @@ Design principles enforced by the engine: 6. Agentic Messages Loop with Dynamic Context - For each turn of the loop (max `agentic_max_turns` turns, default 8): - Update first system message with fresh time/location context - - Send messages to LLM — try native tool calling first (Ollama `tools` API parameter) + - Send messages to LLM — try native tool calling first (`tools` API parameter; Ollama `/api/chat`, or `/chat/completions` when `ollama_base_url` ends in `/v1`, see the transport note in `docs/llm_contexts.md`) - If the model returns HTTP 400 (native tools API not supported), automatically fall back to text-based tool calling for the rest of the session: - Rebuild system message to inject tool descriptions and markdown fence instructions @@ -140,7 +140,7 @@ Design principles enforced by the engine: - LLM returns empty response with no tool calls for multiple turns Tool protocol details: - - Native tool calling (default): Tools are passed to Ollama via the `tools` API parameter in OpenAI-compatible JSON schema format; the LLM requests tools via the standard `tool_calls` field + - Native tool calling (default): Tools are passed to the LLM server (Ollama, or an OpenAI-compatible server such as Strata) via the `tools` API parameter in OpenAI-compatible JSON schema format; the LLM requests tools via the standard `tool_calls` field - Text-based fallback (automatic): If the model returns HTTP 400, the engine switches to injecting tool descriptions as plain text in the system message and parsing `` ```tool_call ``` `` markdown fences from the model's content field - Fallback is detected once per session (first HTTP 400 response) and persists for the rest of the conversation - Internal reasoning uses the `thinking` field (not shown to user) diff --git a/src/jarvis/tools/selection.spec.md b/src/jarvis/tools/selection.spec.md index 89b241c..e7b9691 100644 --- a/src/jarvis/tools/selection.spec.md +++ b/src/jarvis/tools/selection.spec.md @@ -38,7 +38,7 @@ Regardless of strategy, these tools are **always** included: ### Embedding Strategy -1. Embed the user query using `get_embedding()` (calls Ollama `/api/embeddings` with the configured embed model). +1. Embed the user query using `get_embedding()` (calls Ollama `/api/embeddings` with the configured embed model, or `/embeddings` when `ollama_base_url` ends in `/v1`; no embedding means the strategy falls back as below). 2. For each tool (excluding always-included), build a summary string from the tool name (camelCase split) and description, then embed it. 3. Compute cosine similarity between the query embedding and each tool embedding. 4. Select tools using a **relative threshold**: keep tools whose similarity >= `top_score * _RELATIVE_THRESHOLD` (0.97 — nomic-embed-text has a high baseline similarity, so a loose threshold lets the entire catalogue through). diff --git a/tests/test_llm_openai_compat.py b/tests/test_llm_openai_compat.py new file mode 100644 index 0000000..788fa8f --- /dev/null +++ b/tests/test_llm_openai_compat.py @@ -0,0 +1,233 @@ +"""The brain talks to an OpenAI-compatible server (Strata, llama.cpp) as well as Ollama. + +A base URL that ends in ``/v1`` selects the OpenAI-compatible API +(``/chat/completions``, ``/embeddings``). The callers keep the Ollama-shaped +messages and responses they already use, so these tests check what goes over +the wire and what the caller gets back: + +- requests go to ``/chat/completions`` with OpenAI fields (``max_tokens``, + thinking switch, JSON-string tool arguments, no private ``_`` keys), +- responses come back in the Ollama shape the reply engine reads + (``message.content``, ``message.tool_calls[*].function.arguments`` as a dict, + ``message.thinking``), +- streaming, embeddings, the warm-up and the intent judge use the same API. +""" + +from __future__ import annotations + +import json +from unittest.mock import MagicMock, patch + +import pytest + +from jarvis import llm +from jarvis.llm import ( + ToolsNotSupportedError, + call_llm_direct, + call_llm_streaming, + chat_with_messages, + is_openai_compatible, +) + +V1 = "http://127.0.0.1:8080/v1" + + +def _response(body, status=200): + resp = MagicMock() + resp.status_code = status + resp.json.return_value = body + resp.__enter__ = lambda s: s + resp.__exit__ = lambda s, *a: False + if status >= 400: + import requests + + err = requests.exceptions.HTTPError(f"{status}") + err.response = resp + resp.raise_for_status.side_effect = err + else: + resp.raise_for_status.return_value = None + return resp + + +def _completion(content="", tool_calls=None, reasoning=None, finish="stop"): + message = {"role": "assistant", "content": content} + if tool_calls is not None: + message["tool_calls"] = tool_calls + if reasoning is not None: + message["reasoning_content"] = reasoning + return { + "choices": [{"index": 0, "message": message, "finish_reason": finish}], + "usage": {"prompt_tokens": 12, "completion_tokens": 3}, + } + + +def test_v1_suffix_selects_the_openai_api(): + assert is_openai_compatible("http://127.0.0.1:8080/v1") + assert is_openai_compatible("http://127.0.0.1:8080/v1/") + assert not is_openai_compatible("http://127.0.0.1:11434") + assert not is_openai_compatible("") + + +def test_chat_with_messages_returns_tool_calls_in_the_shape_the_engine_reads(): + tc = [{"id": "call_7", "type": "function", + "function": {"name": "getWeather", "arguments": "{\"location\": \"서울\"}"}}] + with patch("jarvis.llm.requests.post", return_value=_response(_completion("", tc, reasoning="hmm"))) as post: + data = chat_with_messages(V1, "qwen", [{"role": "user", "content": "서울 날씨"}], + tools=[{"type": "function", "function": {"name": "getWeather"}}]) + + assert post.call_args.args[0] == f"{V1}/chat/completions" + msg = data["message"] + assert msg["content"] == "" + assert msg["thinking"] == "hmm" + call = msg["tool_calls"][0] + assert call["id"] == "call_7" + assert call["function"]["name"] == "getWeather" + assert call["function"]["arguments"] == {"location": "서울"} + + +def test_tool_call_without_id_gets_one_so_the_tool_result_can_refer_to_it(): + tc = [{"type": "function", "function": {"name": "stop", "arguments": "{}"}}] + with patch("jarvis.llm.requests.post", return_value=_response(_completion("", tc))): + data = chat_with_messages(V1, "qwen", [{"role": "user", "content": "x"}], tools=[{"type": "function"}]) + assert data["message"]["tool_calls"][0]["id"] + + +def test_history_is_sent_as_openai_messages(): + history = [ + {"role": "system", "content": "sys", "_is_tool_guidance": True}, + {"role": "user", "content": "서울 날씨"}, + {"role": "assistant", "content": "", "thinking": "plan", + "tool_calls": [{"id": "call_1", "function": {"name": "getWeather", "arguments": {"location": "서울"}}}]}, + {"role": "tool", "tool_call_id": "call_1", "tool_name": "getWeather", "tool_failed": False, + "content": "맑음"}, + ] + with patch("jarvis.llm.requests.post", return_value=_response(_completion("맑아요"))) as post: + chat_with_messages(V1, "qwen", history, extra_options={"num_predict": 512, "temperature": 0.2}) + + body = post.call_args.kwargs["json"] + sent = body["messages"] + assert sent[0] == {"role": "system", "content": "sys"} + assistant = sent[2] + assert "thinking" not in assistant + call = assistant["tool_calls"][0] + assert call["type"] == "function" + assert json.loads(call["function"]["arguments"]) == {"location": "서울"} + assert sent[3] == {"role": "tool", "tool_call_id": "call_1", "content": "맑음"} + assert body["max_tokens"] == 512 + assert body["temperature"] == 0.2 + assert body["stream"] is False + for ollama_only in ("options", "keep_alive", "think"): + assert ollama_only not in body + # The history the caller keeps is not changed by the conversion. + assert history[2]["tool_calls"][0]["function"]["arguments"] == {"location": "서울"} + + +@pytest.mark.parametrize("thinking", [False, True]) +def test_thinking_switch_reaches_the_chat_template(thinking): + with patch("jarvis.llm.requests.post", return_value=_response(_completion("ok"))) as post: + chat_with_messages(V1, "qwen", [{"role": "user", "content": "x"}], thinking=thinking) + assert post.call_args.kwargs["json"]["chat_template_kwargs"] == {"enable_thinking": thinking} + + +def test_rejected_tools_raise_tools_not_supported(): + with patch("jarvis.llm.requests.post", return_value=_response({"error": "tools"}, status=400)): + with pytest.raises(ToolsNotSupportedError): + chat_with_messages(V1, "qwen", [{"role": "user", "content": "x"}], tools=[{"type": "function"}]) + + +def test_call_llm_direct_returns_the_answer_text(): + with patch("jarvis.llm.requests.post", return_value=_response(_completion("{\"a\": 1}"))) as post: + out = call_llm_direct(V1, "qwen", "system", "user", temperature=0.0) + assert out == "{\"a\": 1}" + body = post.call_args.kwargs["json"] + assert post.call_args.args[0] == f"{V1}/chat/completions" + assert body["messages"] == [{"role": "system", "content": "system"}, {"role": "user", "content": "user"}] + assert body["temperature"] == 0.0 + + +def test_call_llm_direct_returns_none_on_server_error(): + with patch("jarvis.llm.requests.post", return_value=_response({"error": "x"}, status=500)): + assert call_llm_direct(V1, "qwen", "s", "u") is None + + +def _sse(*chunks): + lines = [f"data: {json.dumps(c)}".encode() for c in chunks] + [b"data: [DONE]"] + resp = _response(None) + resp.iter_lines.return_value = lines + return resp + + +def test_streaming_passes_each_piece_of_text_on(): + resp = _sse( + {"choices": [{"delta": {"role": "assistant"}}]}, + {"choices": [{"delta": {"reasoning_content": "think"}}]}, + {"choices": [{"delta": {"content": "안녕"}}]}, + {"choices": [{"delta": {"content": "하세요."}}]}, + {"choices": [], "usage": {"completion_tokens": 2}}, + ) + got = [] + with patch("jarvis.llm.requests.post", return_value=resp) as post: + out = call_llm_streaming(V1, "qwen", "s", "u", on_token=got.append) + assert out == "안녕하세요." + assert got == ["안녕", "하세요."] + body = post.call_args.kwargs["json"] + assert body["stream"] is True + assert post.call_args.kwargs["stream"] is True + + +def test_embeddings_use_the_openai_endpoint(): + from jarvis.memory.embeddings import get_embedding + + with patch("jarvis.memory.embeddings.requests") as req: + req.post.return_value = _response({"data": [{"embedding": [0.5, 0.25]}]}) + vec = get_embedding("hello", V1, "embed-model") + assert vec == [0.5, 0.25] + assert req.post.call_args.args[0] == f"{V1}/embeddings" + assert req.post.call_args.kwargs["json"] == {"model": "embed-model", "input": "hello"} + + +def test_embeddings_fail_open_when_the_server_has_none(): + from jarvis.memory.embeddings import get_embedding + + with patch("jarvis.memory.embeddings.requests") as req: + req.post.return_value = _response({"error": "not found"}, status=404) + assert get_embedding("hello", V1, "embed-model") is None + + +def test_warm_up_asks_for_one_token_through_chat_completions(): + from jarvis.listening.intent_judge import warm_up_ollama_model + + with patch("jarvis.llm.requests.post", return_value=_response(_completion("x"))) as post: + assert warm_up_ollama_model(V1, "qwen", timeout=5) is True + assert post.call_args.args[0] == f"{V1}/chat/completions" + assert post.call_args.kwargs["json"]["max_tokens"] == 1 + + +def test_intent_judge_reads_an_openai_answer(): + from jarvis.listening.intent_judge import IntentJudge, IntentJudgeConfig + from jarvis.listening.transcript_buffer import TranscriptSegment + + answer = json.dumps({"directed": True, "query": "what time is it", "stop": False, + "confidence": "high", "reasoning": "wake word"}) + judge = IntentJudge(IntentJudgeConfig(ollama_base_url=V1, model="qwen")) + seg = TranscriptSegment(text="jarvis what time is it", start_time=1.0, end_time=2.0) + with patch("jarvis.llm.requests.post", return_value=_response(_completion(answer))) as post: + judgment = judge.judge([seg], wake_timestamp=1.0) + assert post.call_args.args[0] == f"{V1}/chat/completions" + assert judgment is not None and judgment.directed and judgment.query == "what time is it" + + +def test_ollama_urls_still_use_the_ollama_api(): + with patch("jarvis.llm.requests.post", return_value=_response({"message": {"content": "ok"}})) as post: + assert call_llm_direct("http://127.0.0.1:11434", "m", "s", "u") == "ok" + assert post.call_args.args[0].endswith("/api/chat") + assert llm.OLLAMA_NUM_CTX == post.call_args.kwargs["json"]["options"]["num_ctx"] + + +def test_dictation_clean_up_uses_chat_completions(): + from jarvis.dictation.dictation_engine import _llm_clean_dictation + + with patch("jarvis.llm.requests.post", return_value=_response(_completion("hello there"))) as post: + out = _llm_clean_dictation("um hello there", V1, model="qwen") + assert out == "hello there" + assert post.call_args.args[0] == f"{V1}/chat/completions"