feat(llm): talk to OpenAI-compatible servers such as Strata

A base URL that ends in /v1 now selects the OpenAI-compatible API: chat goes
to /chat/completions and embeddings to /embeddings. The conversion lives in
src/jarvis/llm.py only, so the reply engine and every other caller keep their
Ollama-shaped messages and responses.

- Requests drop private engine keys, send tool-call arguments as JSON strings,
  map num_predict to max_tokens and the thinking switch to
  chat_template_kwargs.enable_thinking, and leave out num_ctx and keep_alive.
- Responses come back with message.content, tool calls with dict arguments and
  an id, and message.thinking from reasoning_content.
- Streaming, the warm-up (one-token chat completion), the intent judge and the
  dictation clean-up use the same path. A server without an embedding model
  fails open to the text paths.
- Specs and docs/llm_contexts.md describe the transport; roadmap records M3
  step 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-11 12:32:48 +09:00
parent beaacb39cd
commit 6b23a8adc7
11 changed files with 558 additions and 31 deletions

View File

@@ -57,7 +57,8 @@ After transcription, text passes through these stages in order:
1. **Custom dictionary** — case-insensitive whole-word regex replacements
from `dictation_custom_dictionary`. Each entry is `"wrong -> right"`.
2. **LLM filler removal** (optional) — when `dictation_filler_removal` is
enabled, sends the text to the local Ollama instance (same model as the
enabled, sends the text to the local LLM server (Ollama, or the
OpenAI-compatible `/chat/completions` when the URL ends in `/v1`; same model as the
assistant) with a prompt to remove filler words (um, uh, like, you know,
etc.) while preserving meaning. Uses a 5-second timeout; falls back to the
unprocessed text on failure.

View File

@@ -450,6 +450,15 @@ def _llm_clean_dictation(text: str, ollama_base_url: str, model: str = "gemma4:e
f"{text}"
)
from ..llm import call_llm_direct, is_openai_compatible
if is_openai_compatible(ollama_base_url):
cleaned = (call_llm_direct(ollama_base_url, model, "", prompt, timeout_sec=5, thinking=thinking) or "").strip()
if cleaned:
debug_log(f"LLM filler removal: {text!r} → {cleaned!r}", "dictation")
return cleaned
return text
try:
resp = requests.post(
f"{ollama_base_url}/api/generate",

View File

@@ -13,6 +13,7 @@ from dataclasses import dataclass
from typing import Optional, List
from ..debug import debug_log
from ..llm import openai_chat_body, extract_text_from_response, is_openai_compatible, post_chat_completion
from .transcript_buffer import TranscriptSegment
try:
@@ -32,6 +33,18 @@ def warm_up_ollama_model(base_url: str, model: str, timeout: float) -> bool:
"""
if not REQUESTS_AVAILABLE or not base_url or not model:
return False
if is_openai_compatible(base_url):
# The server loads its model at launch; one token checks it answers.
body = openai_chat_body(model, [{"role": "user", "content": "hi"}], stream=False,
thinking=False, max_tokens=1)
try:
response = post_chat_completion(base_url, body, timeout)
ok = response.status_code == 200
debug_log(f"llm warmup {'ok' if ok else f'failed HTTP {response.status_code}'} (model={model})", "voice")
return ok
except Exception as e:
debug_log(f"llm warmup error (model={model}): {e}", "voice")
return False
try:
response = requests.post(
f"{base_url}/api/generate",
@@ -418,42 +431,56 @@ Examples:
# (which was the original timeout culprit). Ollama's default is
# 5m; we pin to 30m because voice sessions can have long quiet
# stretches and reloading mid-conversation is a bad experience.
response = requests.post(
f"{self.config.ollama_base_url}/api/generate",
json={
"model": self.config.model,
"prompt": user_prompt,
"system": system_prompt,
"stream": False,
"think": self.config.thinking,
"keep_alive": "30m",
"options": {
"temperature": 0.0,
"num_predict": 200,
# Headroom for: ~2k-token system prompt + up to 2 minutes
# of chatty multi-speaker transcript (default
# transcript_buffer_duration_sec=120 in listener.py).
# 4096 was cutting close to 90% utilisation in the
# worst case after the prompt grew in PR #362, which
# risks silent ollama truncation of the system
# prompt's tail.
"num_ctx": 8192,
if is_openai_compatible(self.config.ollama_base_url):
response = post_chat_completion(
self.config.ollama_base_url,
openai_chat_body(
self.config.model,
[{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}],
stream=False, thinking=self.config.thinking, max_tokens=200, temperature=0.0,
),
self.config.timeout_sec,
)
else:
response = requests.post(
f"{self.config.ollama_base_url}/api/generate",
json={
"model": self.config.model,
"prompt": user_prompt,
"system": system_prompt,
"stream": False,
"think": self.config.thinking,
"keep_alive": "30m",
"options": {
"temperature": 0.0,
"num_predict": 200,
# Headroom for: ~2k-token system prompt + up to 2 minutes
# of chatty multi-speaker transcript (default
# transcript_buffer_duration_sec=120 in listener.py).
# 4096 was cutting close to 90% utilisation in the
# worst case after the prompt grew in PR #362, which
# risks silent ollama truncation of the system
# prompt's tail.
"num_ctx": 8192,
},
},
},
timeout=self.config.timeout_sec,
)
timeout=self.config.timeout_sec,
)
if response.status_code != 200:
# Don't back off on transient HTTP errors — voice is high-turn
# and a 503 from an overloaded Ollama shouldn't kill the next
# 30s of intent judging. Retry on the next engagement signal.
reason = f"HTTP {response.status_code} from Ollama"
reason = f"HTTP {response.status_code} from the LLM server"
debug_log(f"intent judge: {reason}", "voice")
self._last_failure_reason = reason
return None
result = response.json()
response_text = result.get("response", "")
if is_openai_compatible(self.config.ollama_base_url):
response_text = extract_text_from_response(result) or ""
else:
response_text = result.get("response", "")
judgment = self._parse_response(response_text)

View File

@@ -114,7 +114,7 @@ On small models, a caveat line is appended above a more involved example to set
**What gets warmed:**
- **Whisper** — loading the model; additionally a silent-audio transcribe so the first real utterance doesn't pay the cold-decode cost. Both the MLX and faster-whisper backends do this.
- **Chat model** (`cfg.ollama_chat_model`) — a minimal Ollama `/api/generate` request with `keep_alive=30m` so the weights stay resident.
- **Chat model** (`cfg.ollama_chat_model`) — a minimal Ollama `/api/generate` request with `keep_alive=30m` so the weights stay resident. When `ollama_base_url` ends in `/v1` (an OpenAI-compatible server such as Strata, which loads its model at launch) the warm-up is a one-token `/chat/completions` request that only checks the server answers, and the intent judge itself also calls `/chat/completions`.
- **Intent judge model** (`cfg.intent_judge_model`) — same pattern. If it points at the same Ollama model as the chat model, a single warmup covers both roles (Ollama loads the weights once).
**Concurrency:** LLM warmups run in daemon threads started before Whisper loads, so they overlap with Whisper initialisation. After Whisper finishes, the listener joins the warmup threads with a **single 60 s budget** shared across them all. If the budget is exhausted, the listener continues (with a `⏳ Some models still warming — continuing anyway` notice) and the first engagement pays the cold-load cost on demand.

View File

@@ -6,6 +6,8 @@ import os
import sys
import requests
import json
import time
import uuid
def _caller_name() -> str:
@@ -35,6 +37,152 @@ class ToolsNotSupportedError(Exception):
pass
# ---------------------------------------------------------------------------
# OpenAI-compatible servers (Strata, llama.cpp, Ollama's own /v1)
# ---------------------------------------------------------------------------
#
# A base URL that ends in ``/v1`` selects the OpenAI chat-completions API.
# Callers keep building Ollama-shaped messages and reading Ollama-shaped
# responses (``message.content``, ``message.tool_calls`` with dict arguments,
# ``message.thinking``); the conversion lives here only. The server owns the
# context size, so ``num_ctx`` and ``keep_alive`` are not sent.
_OPENAI_MESSAGE_KEYS = ("role", "content", "tool_calls", "tool_call_id", "name")
def is_openai_compatible(base_url: str) -> bool:
"""True when ``base_url`` points at an OpenAI-compatible API (ends in ``/v1``)."""
return bool(base_url) and base_url.rstrip("/").endswith("/v1")
def _openai_messages(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Copy Ollama-shaped messages into OpenAI ones without changing the originals.
Private engine keys (``_is_tool_guidance``), ``thinking``, ``tool_name`` and
``tool_failed`` are dropped, and tool-call arguments become JSON strings.
"""
out: List[Dict[str, Any]] = []
for m in messages:
msg = {k: m[k] for k in _OPENAI_MESSAGE_KEYS if k in m}
if msg.get("content") is None:
msg["content"] = ""
calls = m.get("tool_calls")
if calls:
converted = []
for i, tc in enumerate(calls):
fn = tc.get("function") or {}
args = fn.get("arguments")
if not isinstance(args, str):
args = json.dumps(args or {}, ensure_ascii=False)
converted.append({
"id": tc.get("id") or f"call_{i}",
"type": "function",
"function": {"name": fn.get("name", ""), "arguments": args},
})
msg["tool_calls"] = converted
else:
msg.pop("tool_calls", None)
out.append(msg)
return out
def openai_chat_body(
chat_model: str,
messages: List[Dict[str, Any]],
*,
stream: bool,
thinking: bool,
max_tokens: Optional[int] = None,
temperature: Optional[float] = None,
tools: Optional[List[Dict[str, Any]]] = None,
) -> Dict[str, Any]:
body: Dict[str, Any] = {
"model": chat_model,
"messages": _openai_messages(messages),
"stream": stream,
# Qwen-family templates think unless told not to; spoken answers must not.
"chat_template_kwargs": {"enable_thinking": bool(thinking)},
}
if max_tokens is not None and max_tokens > 0:
body["max_tokens"] = int(max_tokens)
if temperature is not None:
body["temperature"] = temperature
if tools:
body["tools"] = tools
return body
def post_chat_completion(base_url: str, body: Dict[str, Any], timeout_sec: float, stream: bool = False):
"""POST ``body`` to ``<base_url>/chat/completions`` and return the response."""
return requests.post(
f"{base_url.rstrip('/')}/chat/completions", json=body, timeout=timeout_sec, stream=stream
)
def _parse_tool_arguments(raw: Any) -> Dict[str, Any]:
if isinstance(raw, dict):
return raw
if isinstance(raw, str) and raw.strip():
try:
parsed = json.loads(raw)
if isinstance(parsed, dict):
return parsed
except json.JSONDecodeError:
pass
debug_log(f"tool-call arguments are not a JSON object (len={len(raw)})", "llm")
return {}
def _ollama_shaped(data: Dict[str, Any]) -> Optional[Dict[str, Any]]:
"""Turn an OpenAI chat completion into the Ollama response shape callers read."""
choices = data.get("choices") if isinstance(data, dict) else None
if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict):
return None
choice = choices[0]
src = choice.get("message") or {}
msg: Dict[str, Any] = {"role": "assistant", "content": src.get("content") or ""}
reasoning = src.get("reasoning_content") or src.get("reasoning")
if reasoning:
msg["thinking"] = reasoning
calls = []
for tc in src.get("tool_calls") or []:
fn = tc.get("function") or {}
calls.append({
"id": tc.get("id") or f"call_{uuid.uuid4().hex[:8]}",
"type": "function",
"function": {"name": fn.get("name", ""), "arguments": _parse_tool_arguments(fn.get("arguments"))},
})
if calls:
msg["tool_calls"] = calls
return {"message": msg, "done": True, "done_reason": choice.get("finish_reason"), "usage": data.get("usage")}
def _log_openai_timing(data: Dict[str, Any], elapsed: float, caller: str) -> None:
"""Per-call latency line for OpenAI-compatible servers (no server-side phase split)."""
try:
usage = data.get("usage") or {}
print(
f" ⏱️ llm[{caller}] total={elapsed:.2f}s "
f"prompt={usage.get('prompt_tokens')}t gen={usage.get('completion_tokens')}t",
flush=True,
)
except Exception: # pragma: no cover - logging must never break a reply
pass
def _openai_chat(base_url: str, body: Dict[str, Any], timeout_sec: float, caller: str) -> Dict[str, Any]:
"""Non-streaming chat completion, returned in the Ollama shape. Raises requests errors."""
t0 = time.perf_counter()
with post_chat_completion(base_url, body, timeout_sec) as resp:
resp.raise_for_status()
data = resp.json()
_log_openai_timing(data, time.perf_counter() - t0, caller)
shaped = _ollama_shaped(data)
if shaped is None:
raise ValueError(f"no choices in response keys={list(data.keys()) if isinstance(data, dict) else type(data)}")
return shaped
def _log_ollama_timing(data: Dict[str, Any], num_ctx: int, caller: str) -> None:
"""Emit a one-line per-call latency breakdown so a slow voice turn can be
attributed to a specific internal LLM call (router / enrichment / digest /
@@ -81,6 +229,20 @@ def call_llm_direct(base_url: str, chat_model: str, system_prompt: str, user_con
{"role": "user", "content": user_content}
]
if is_openai_compatible(base_url):
caller = _caller_name()
body = openai_chat_body(chat_model, messages, stream=False, thinking=thinking, temperature=temperature)
try:
content = _openai_chat(base_url, body, timeout_sec, caller)["message"]["content"]
if content.strip():
return content
debug_log("call_llm_direct: empty content from OpenAI-compatible server", "llm")
except requests.exceptions.Timeout:
debug_log(f"call_llm_direct: timeout after {timeout_sec}s", "llm")
except Exception as e:
debug_log(f"call_llm_direct: request failed — {e}", "llm")
return None
options: Dict[str, Any] = {"num_ctx": num_ctx}
if temperature is not None:
options["temperature"] = temperature
@@ -150,6 +312,9 @@ def call_llm_streaming(
{"role": "user", "content": user_content}
]
if is_openai_compatible(base_url):
return _openai_stream(base_url, chat_model, messages, on_token, timeout_sec, thinking)
payload: Dict[str, Any] = {
"model": chat_model,
"messages": messages,
@@ -197,6 +362,49 @@ def call_llm_streaming(
return None
def _openai_stream(
base_url: str,
chat_model: str,
messages: List[Dict[str, Any]],
on_token: Optional[Callable[[str], None]],
timeout_sec: float,
thinking: bool,
) -> Optional[str]:
"""Server-sent-events stream from ``/chat/completions``; answer text only (no reasoning)."""
body = openai_chat_body(chat_model, messages, stream=True, thinking=thinking)
try:
with post_chat_completion(base_url, body, timeout_sec, stream=True) as resp:
resp.raise_for_status()
full_response = []
for line in resp.iter_lines():
if not line:
continue
text = line.decode("utf-8", "replace") if isinstance(line, bytes) else line
if not text.startswith("data:"):
continue
payload = text[5:].strip()
if payload == "[DONE]":
break
try:
chunk = json.loads(payload)
except json.JSONDecodeError:
continue
for choice in chunk.get("choices") or []:
content = (choice.get("delta") or {}).get("content")
if content:
full_response.append(content)
if on_token:
on_token(content)
result = "".join(full_response)
return result if result.strip() else None
except requests.exceptions.Timeout:
debug_log(f"call_llm_streaming: timeout after {timeout_sec}s", "llm")
return None
except Exception as e:
debug_log(f"call_llm_streaming: request failed — {e}", "llm")
return None
def extract_text_from_response(data: Dict[str, Any]) -> Optional[str]:
"""Extract text from LLM response - supports multiple response formats."""
# Preferred: Ollama chat non-stream format
@@ -251,6 +459,29 @@ def chat_with_messages(
Returns the parsed JSON response dict on success, or None on error/timeout.
"""
if is_openai_compatible(base_url):
opts = extra_options if isinstance(extra_options, dict) else {}
body = openai_chat_body(
chat_model, messages, stream=False, thinking=thinking,
max_tokens=opts.get("num_predict"), temperature=opts.get("temperature"), tools=tools,
)
caller = _caller_name()
try:
return _openai_chat(base_url, body, timeout_sec, caller)
except requests.exceptions.Timeout:
print(" ⏱️ LLM request timed out", flush=True)
except requests.exceptions.ConnectionError as e:
print(f" ❌ LLM connection error: {e}", flush=True)
except requests.exceptions.HTTPError as e:
if e.response is not None and e.response.status_code == 400 and tools:
raise ToolsNotSupportedError(
f"Model {chat_model!r} returned HTTP 400 — native tools API not supported"
)
print(f" ❌ LLM HTTP error: {e}", flush=True)
except Exception as e:
print(f" ❌ LLM error: {e}", flush=True)
return None
# Main agentic chat uses 8192 so the system prompt (tool list + protocol
# guidance + memory context) doesn't overflow and force ollama to truncate
# — which previously dropped the tool schema on smaller models like

View File

@@ -2,7 +2,31 @@ from __future__ import annotations
import requests
def _openai_embedding(text: str, base_url: str, model: str, timeout_sec: float) -> list[float] | None:
"""``/embeddings`` of an OpenAI-compatible server. A server without an
embedding model (Strata serves the chat model only) answers an error, and
the caller falls back to text search."""
try:
resp = requests.post(
f"{base_url.rstrip('/')}/embeddings",
json={"model": model, "input": text},
timeout=timeout_sec,
)
resp.raise_for_status()
data = resp.json().get("data") or []
vec = data[0].get("embedding") if data and isinstance(data[0], dict) else None
if isinstance(vec, list):
return [float(x) for x in vec]
except Exception:
return None
return None
def get_embedding(text: str, base_url: str, model: str, timeout_sec: float = 15.0) -> list[float] | None:
from ..llm import is_openai_compatible
if is_openai_compatible(base_url):
return _openai_embedding(text, base_url, model, timeout_sec)
try:
resp = requests.post(
f"{base_url.rstrip('/')}/api/embeddings",

View File

@@ -77,7 +77,7 @@ Design principles enforced by the engine:
6. Agentic Messages Loop with Dynamic Context
- For each turn of the loop (max `agentic_max_turns` turns, default 8):
- Update first system message with fresh time/location context
- Send messages to LLM — try native tool calling first (Ollama `tools` API parameter)
- Send messages to LLM — try native tool calling first (`tools` API parameter; Ollama `/api/chat`, or `/chat/completions` when `ollama_base_url` ends in `/v1`, see the transport note in `docs/llm_contexts.md`)
- If the model returns HTTP 400 (native tools API not supported), automatically fall back
to text-based tool calling for the rest of the session:
- Rebuild system message to inject tool descriptions and markdown fence instructions
@@ -140,7 +140,7 @@ Design principles enforced by the engine:
- LLM returns empty response with no tool calls for multiple turns
Tool protocol details:
- Native tool calling (default): Tools are passed to Ollama via the `tools` API parameter in OpenAI-compatible JSON schema format; the LLM requests tools via the standard `tool_calls` field
- Native tool calling (default): Tools are passed to the LLM server (Ollama, or an OpenAI-compatible server such as Strata) via the `tools` API parameter in OpenAI-compatible JSON schema format; the LLM requests tools via the standard `tool_calls` field
- Text-based fallback (automatic): If the model returns HTTP 400, the engine switches to injecting tool descriptions as plain text in the system message and parsing `` ```tool_call ``` `` markdown fences from the model's content field
- Fallback is detected once per session (first HTTP 400 response) and persists for the rest of the conversation
- Internal reasoning uses the `thinking` field (not shown to user)

View File

@@ -38,7 +38,7 @@ Regardless of strategy, these tools are **always** included:
### Embedding Strategy
1. Embed the user query using `get_embedding()` (calls Ollama `/api/embeddings` with the configured embed model).
1. Embed the user query using `get_embedding()` (calls Ollama `/api/embeddings` with the configured embed model, or `/embeddings` when `ollama_base_url` ends in `/v1`; no embedding means the strategy falls back as below).
2. For each tool (excluding always-included), build a summary string from the tool name (camelCase split) and description, then embed it.
3. Compute cosine similarity between the query embedding and each tool embedding.
4. Select tools using a **relative threshold**: keep tools whose similarity >= `top_score * _RELATIVE_THRESHOLD` (0.97 — nomic-embed-text has a high baseline similarity, so a loose threshold lets the entire catalogue through).