feat(llm): talk to OpenAI-compatible servers such as Strata
A base URL that ends in /v1 now selects the OpenAI-compatible API: chat goes to /chat/completions and embeddings to /embeddings. The conversion lives in src/jarvis/llm.py only, so the reply engine and every other caller keep their Ollama-shaped messages and responses. - Requests drop private engine keys, send tool-call arguments as JSON strings, map num_predict to max_tokens and the thinking switch to chat_template_kwargs.enable_thinking, and leave out num_ctx and keep_alive. - Responses come back with message.content, tool calls with dict arguments and an id, and message.thinking from reasoning_content. - Streaming, the warm-up (one-token chat completion), the intent judge and the dictation clean-up use the same path. A server without an embedding model fails open to the text paths. - Specs and docs/llm_contexts.md describe the transport; roadmap records M3 step 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -57,7 +57,8 @@ After transcription, text passes through these stages in order:
|
||||
1. **Custom dictionary** — case-insensitive whole-word regex replacements
|
||||
from `dictation_custom_dictionary`. Each entry is `"wrong -> right"`.
|
||||
2. **LLM filler removal** (optional) — when `dictation_filler_removal` is
|
||||
enabled, sends the text to the local Ollama instance (same model as the
|
||||
enabled, sends the text to the local LLM server (Ollama, or the
|
||||
OpenAI-compatible `/chat/completions` when the URL ends in `/v1`; same model as the
|
||||
assistant) with a prompt to remove filler words (um, uh, like, you know,
|
||||
etc.) while preserving meaning. Uses a 5-second timeout; falls back to the
|
||||
unprocessed text on failure.
|
||||
|
||||
@@ -450,6 +450,15 @@ def _llm_clean_dictation(text: str, ollama_base_url: str, model: str = "gemma4:e
|
||||
f"{text}"
|
||||
)
|
||||
|
||||
from ..llm import call_llm_direct, is_openai_compatible
|
||||
|
||||
if is_openai_compatible(ollama_base_url):
|
||||
cleaned = (call_llm_direct(ollama_base_url, model, "", prompt, timeout_sec=5, thinking=thinking) or "").strip()
|
||||
if cleaned:
|
||||
debug_log(f"LLM filler removal: {text!r} → {cleaned!r}", "dictation")
|
||||
return cleaned
|
||||
return text
|
||||
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{ollama_base_url}/api/generate",
|
||||
|
||||
@@ -13,6 +13,7 @@ from dataclasses import dataclass
|
||||
from typing import Optional, List
|
||||
|
||||
from ..debug import debug_log
|
||||
from ..llm import openai_chat_body, extract_text_from_response, is_openai_compatible, post_chat_completion
|
||||
from .transcript_buffer import TranscriptSegment
|
||||
|
||||
try:
|
||||
@@ -32,6 +33,18 @@ def warm_up_ollama_model(base_url: str, model: str, timeout: float) -> bool:
|
||||
"""
|
||||
if not REQUESTS_AVAILABLE or not base_url or not model:
|
||||
return False
|
||||
if is_openai_compatible(base_url):
|
||||
# The server loads its model at launch; one token checks it answers.
|
||||
body = openai_chat_body(model, [{"role": "user", "content": "hi"}], stream=False,
|
||||
thinking=False, max_tokens=1)
|
||||
try:
|
||||
response = post_chat_completion(base_url, body, timeout)
|
||||
ok = response.status_code == 200
|
||||
debug_log(f"llm warmup {'ok' if ok else f'failed HTTP {response.status_code}'} (model={model})", "voice")
|
||||
return ok
|
||||
except Exception as e:
|
||||
debug_log(f"llm warmup error (model={model}): {e}", "voice")
|
||||
return False
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{base_url}/api/generate",
|
||||
@@ -418,42 +431,56 @@ Examples:
|
||||
# (which was the original timeout culprit). Ollama's default is
|
||||
# 5m; we pin to 30m because voice sessions can have long quiet
|
||||
# stretches and reloading mid-conversation is a bad experience.
|
||||
response = requests.post(
|
||||
f"{self.config.ollama_base_url}/api/generate",
|
||||
json={
|
||||
"model": self.config.model,
|
||||
"prompt": user_prompt,
|
||||
"system": system_prompt,
|
||||
"stream": False,
|
||||
"think": self.config.thinking,
|
||||
"keep_alive": "30m",
|
||||
"options": {
|
||||
"temperature": 0.0,
|
||||
"num_predict": 200,
|
||||
# Headroom for: ~2k-token system prompt + up to 2 minutes
|
||||
# of chatty multi-speaker transcript (default
|
||||
# transcript_buffer_duration_sec=120 in listener.py).
|
||||
# 4096 was cutting close to 90% utilisation in the
|
||||
# worst case after the prompt grew in PR #362, which
|
||||
# risks silent ollama truncation of the system
|
||||
# prompt's tail.
|
||||
"num_ctx": 8192,
|
||||
if is_openai_compatible(self.config.ollama_base_url):
|
||||
response = post_chat_completion(
|
||||
self.config.ollama_base_url,
|
||||
openai_chat_body(
|
||||
self.config.model,
|
||||
[{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}],
|
||||
stream=False, thinking=self.config.thinking, max_tokens=200, temperature=0.0,
|
||||
),
|
||||
self.config.timeout_sec,
|
||||
)
|
||||
else:
|
||||
response = requests.post(
|
||||
f"{self.config.ollama_base_url}/api/generate",
|
||||
json={
|
||||
"model": self.config.model,
|
||||
"prompt": user_prompt,
|
||||
"system": system_prompt,
|
||||
"stream": False,
|
||||
"think": self.config.thinking,
|
||||
"keep_alive": "30m",
|
||||
"options": {
|
||||
"temperature": 0.0,
|
||||
"num_predict": 200,
|
||||
# Headroom for: ~2k-token system prompt + up to 2 minutes
|
||||
# of chatty multi-speaker transcript (default
|
||||
# transcript_buffer_duration_sec=120 in listener.py).
|
||||
# 4096 was cutting close to 90% utilisation in the
|
||||
# worst case after the prompt grew in PR #362, which
|
||||
# risks silent ollama truncation of the system
|
||||
# prompt's tail.
|
||||
"num_ctx": 8192,
|
||||
},
|
||||
},
|
||||
},
|
||||
timeout=self.config.timeout_sec,
|
||||
)
|
||||
timeout=self.config.timeout_sec,
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
# Don't back off on transient HTTP errors — voice is high-turn
|
||||
# and a 503 from an overloaded Ollama shouldn't kill the next
|
||||
# 30s of intent judging. Retry on the next engagement signal.
|
||||
reason = f"HTTP {response.status_code} from Ollama"
|
||||
reason = f"HTTP {response.status_code} from the LLM server"
|
||||
debug_log(f"intent judge: {reason}", "voice")
|
||||
self._last_failure_reason = reason
|
||||
return None
|
||||
|
||||
result = response.json()
|
||||
response_text = result.get("response", "")
|
||||
if is_openai_compatible(self.config.ollama_base_url):
|
||||
response_text = extract_text_from_response(result) or ""
|
||||
else:
|
||||
response_text = result.get("response", "")
|
||||
|
||||
judgment = self._parse_response(response_text)
|
||||
|
||||
|
||||
@@ -114,7 +114,7 @@ On small models, a caveat line is appended above a more involved example to set
|
||||
|
||||
**What gets warmed:**
|
||||
- **Whisper** — loading the model; additionally a silent-audio transcribe so the first real utterance doesn't pay the cold-decode cost. Both the MLX and faster-whisper backends do this.
|
||||
- **Chat model** (`cfg.ollama_chat_model`) — a minimal Ollama `/api/generate` request with `keep_alive=30m` so the weights stay resident.
|
||||
- **Chat model** (`cfg.ollama_chat_model`) — a minimal Ollama `/api/generate` request with `keep_alive=30m` so the weights stay resident. When `ollama_base_url` ends in `/v1` (an OpenAI-compatible server such as Strata, which loads its model at launch) the warm-up is a one-token `/chat/completions` request that only checks the server answers, and the intent judge itself also calls `/chat/completions`.
|
||||
- **Intent judge model** (`cfg.intent_judge_model`) — same pattern. If it points at the same Ollama model as the chat model, a single warmup covers both roles (Ollama loads the weights once).
|
||||
|
||||
**Concurrency:** LLM warmups run in daemon threads started before Whisper loads, so they overlap with Whisper initialisation. After Whisper finishes, the listener joins the warmup threads with a **single 60 s budget** shared across them all. If the budget is exhausted, the listener continues (with a `⏳ Some models still warming — continuing anyway` notice) and the first engagement pays the cold-load cost on demand.
|
||||
|
||||
@@ -6,6 +6,8 @@ import os
|
||||
import sys
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
|
||||
|
||||
def _caller_name() -> str:
|
||||
@@ -35,6 +37,152 @@ class ToolsNotSupportedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OpenAI-compatible servers (Strata, llama.cpp, Ollama's own /v1)
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# A base URL that ends in ``/v1`` selects the OpenAI chat-completions API.
|
||||
# Callers keep building Ollama-shaped messages and reading Ollama-shaped
|
||||
# responses (``message.content``, ``message.tool_calls`` with dict arguments,
|
||||
# ``message.thinking``); the conversion lives here only. The server owns the
|
||||
# context size, so ``num_ctx`` and ``keep_alive`` are not sent.
|
||||
|
||||
_OPENAI_MESSAGE_KEYS = ("role", "content", "tool_calls", "tool_call_id", "name")
|
||||
|
||||
|
||||
def is_openai_compatible(base_url: str) -> bool:
|
||||
"""True when ``base_url`` points at an OpenAI-compatible API (ends in ``/v1``)."""
|
||||
return bool(base_url) and base_url.rstrip("/").endswith("/v1")
|
||||
|
||||
|
||||
def _openai_messages(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""Copy Ollama-shaped messages into OpenAI ones without changing the originals.
|
||||
|
||||
Private engine keys (``_is_tool_guidance``), ``thinking``, ``tool_name`` and
|
||||
``tool_failed`` are dropped, and tool-call arguments become JSON strings.
|
||||
"""
|
||||
out: List[Dict[str, Any]] = []
|
||||
for m in messages:
|
||||
msg = {k: m[k] for k in _OPENAI_MESSAGE_KEYS if k in m}
|
||||
if msg.get("content") is None:
|
||||
msg["content"] = ""
|
||||
calls = m.get("tool_calls")
|
||||
if calls:
|
||||
converted = []
|
||||
for i, tc in enumerate(calls):
|
||||
fn = tc.get("function") or {}
|
||||
args = fn.get("arguments")
|
||||
if not isinstance(args, str):
|
||||
args = json.dumps(args or {}, ensure_ascii=False)
|
||||
converted.append({
|
||||
"id": tc.get("id") or f"call_{i}",
|
||||
"type": "function",
|
||||
"function": {"name": fn.get("name", ""), "arguments": args},
|
||||
})
|
||||
msg["tool_calls"] = converted
|
||||
else:
|
||||
msg.pop("tool_calls", None)
|
||||
out.append(msg)
|
||||
return out
|
||||
|
||||
|
||||
def openai_chat_body(
|
||||
chat_model: str,
|
||||
messages: List[Dict[str, Any]],
|
||||
*,
|
||||
stream: bool,
|
||||
thinking: bool,
|
||||
max_tokens: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
tools: Optional[List[Dict[str, Any]]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
body: Dict[str, Any] = {
|
||||
"model": chat_model,
|
||||
"messages": _openai_messages(messages),
|
||||
"stream": stream,
|
||||
# Qwen-family templates think unless told not to; spoken answers must not.
|
||||
"chat_template_kwargs": {"enable_thinking": bool(thinking)},
|
||||
}
|
||||
if max_tokens is not None and max_tokens > 0:
|
||||
body["max_tokens"] = int(max_tokens)
|
||||
if temperature is not None:
|
||||
body["temperature"] = temperature
|
||||
if tools:
|
||||
body["tools"] = tools
|
||||
return body
|
||||
|
||||
|
||||
def post_chat_completion(base_url: str, body: Dict[str, Any], timeout_sec: float, stream: bool = False):
|
||||
"""POST ``body`` to ``<base_url>/chat/completions`` and return the response."""
|
||||
return requests.post(
|
||||
f"{base_url.rstrip('/')}/chat/completions", json=body, timeout=timeout_sec, stream=stream
|
||||
)
|
||||
|
||||
|
||||
def _parse_tool_arguments(raw: Any) -> Dict[str, Any]:
|
||||
if isinstance(raw, dict):
|
||||
return raw
|
||||
if isinstance(raw, str) and raw.strip():
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
debug_log(f"tool-call arguments are not a JSON object (len={len(raw)})", "llm")
|
||||
return {}
|
||||
|
||||
|
||||
def _ollama_shaped(data: Dict[str, Any]) -> Optional[Dict[str, Any]]:
|
||||
"""Turn an OpenAI chat completion into the Ollama response shape callers read."""
|
||||
choices = data.get("choices") if isinstance(data, dict) else None
|
||||
if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict):
|
||||
return None
|
||||
choice = choices[0]
|
||||
src = choice.get("message") or {}
|
||||
msg: Dict[str, Any] = {"role": "assistant", "content": src.get("content") or ""}
|
||||
reasoning = src.get("reasoning_content") or src.get("reasoning")
|
||||
if reasoning:
|
||||
msg["thinking"] = reasoning
|
||||
calls = []
|
||||
for tc in src.get("tool_calls") or []:
|
||||
fn = tc.get("function") or {}
|
||||
calls.append({
|
||||
"id": tc.get("id") or f"call_{uuid.uuid4().hex[:8]}",
|
||||
"type": "function",
|
||||
"function": {"name": fn.get("name", ""), "arguments": _parse_tool_arguments(fn.get("arguments"))},
|
||||
})
|
||||
if calls:
|
||||
msg["tool_calls"] = calls
|
||||
return {"message": msg, "done": True, "done_reason": choice.get("finish_reason"), "usage": data.get("usage")}
|
||||
|
||||
|
||||
def _log_openai_timing(data: Dict[str, Any], elapsed: float, caller: str) -> None:
|
||||
"""Per-call latency line for OpenAI-compatible servers (no server-side phase split)."""
|
||||
try:
|
||||
usage = data.get("usage") or {}
|
||||
print(
|
||||
f" ⏱️ llm[{caller}] total={elapsed:.2f}s "
|
||||
f"prompt={usage.get('prompt_tokens')}t gen={usage.get('completion_tokens')}t",
|
||||
flush=True,
|
||||
)
|
||||
except Exception: # pragma: no cover - logging must never break a reply
|
||||
pass
|
||||
|
||||
|
||||
def _openai_chat(base_url: str, body: Dict[str, Any], timeout_sec: float, caller: str) -> Dict[str, Any]:
|
||||
"""Non-streaming chat completion, returned in the Ollama shape. Raises requests errors."""
|
||||
t0 = time.perf_counter()
|
||||
with post_chat_completion(base_url, body, timeout_sec) as resp:
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
_log_openai_timing(data, time.perf_counter() - t0, caller)
|
||||
shaped = _ollama_shaped(data)
|
||||
if shaped is None:
|
||||
raise ValueError(f"no choices in response keys={list(data.keys()) if isinstance(data, dict) else type(data)}")
|
||||
return shaped
|
||||
|
||||
|
||||
def _log_ollama_timing(data: Dict[str, Any], num_ctx: int, caller: str) -> None:
|
||||
"""Emit a one-line per-call latency breakdown so a slow voice turn can be
|
||||
attributed to a specific internal LLM call (router / enrichment / digest /
|
||||
@@ -81,6 +229,20 @@ def call_llm_direct(base_url: str, chat_model: str, system_prompt: str, user_con
|
||||
{"role": "user", "content": user_content}
|
||||
]
|
||||
|
||||
if is_openai_compatible(base_url):
|
||||
caller = _caller_name()
|
||||
body = openai_chat_body(chat_model, messages, stream=False, thinking=thinking, temperature=temperature)
|
||||
try:
|
||||
content = _openai_chat(base_url, body, timeout_sec, caller)["message"]["content"]
|
||||
if content.strip():
|
||||
return content
|
||||
debug_log("call_llm_direct: empty content from OpenAI-compatible server", "llm")
|
||||
except requests.exceptions.Timeout:
|
||||
debug_log(f"call_llm_direct: timeout after {timeout_sec}s", "llm")
|
||||
except Exception as e:
|
||||
debug_log(f"call_llm_direct: request failed — {e}", "llm")
|
||||
return None
|
||||
|
||||
options: Dict[str, Any] = {"num_ctx": num_ctx}
|
||||
if temperature is not None:
|
||||
options["temperature"] = temperature
|
||||
@@ -150,6 +312,9 @@ def call_llm_streaming(
|
||||
{"role": "user", "content": user_content}
|
||||
]
|
||||
|
||||
if is_openai_compatible(base_url):
|
||||
return _openai_stream(base_url, chat_model, messages, on_token, timeout_sec, thinking)
|
||||
|
||||
payload: Dict[str, Any] = {
|
||||
"model": chat_model,
|
||||
"messages": messages,
|
||||
@@ -197,6 +362,49 @@ def call_llm_streaming(
|
||||
return None
|
||||
|
||||
|
||||
def _openai_stream(
|
||||
base_url: str,
|
||||
chat_model: str,
|
||||
messages: List[Dict[str, Any]],
|
||||
on_token: Optional[Callable[[str], None]],
|
||||
timeout_sec: float,
|
||||
thinking: bool,
|
||||
) -> Optional[str]:
|
||||
"""Server-sent-events stream from ``/chat/completions``; answer text only (no reasoning)."""
|
||||
body = openai_chat_body(chat_model, messages, stream=True, thinking=thinking)
|
||||
try:
|
||||
with post_chat_completion(base_url, body, timeout_sec, stream=True) as resp:
|
||||
resp.raise_for_status()
|
||||
full_response = []
|
||||
for line in resp.iter_lines():
|
||||
if not line:
|
||||
continue
|
||||
text = line.decode("utf-8", "replace") if isinstance(line, bytes) else line
|
||||
if not text.startswith("data:"):
|
||||
continue
|
||||
payload = text[5:].strip()
|
||||
if payload == "[DONE]":
|
||||
break
|
||||
try:
|
||||
chunk = json.loads(payload)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
for choice in chunk.get("choices") or []:
|
||||
content = (choice.get("delta") or {}).get("content")
|
||||
if content:
|
||||
full_response.append(content)
|
||||
if on_token:
|
||||
on_token(content)
|
||||
result = "".join(full_response)
|
||||
return result if result.strip() else None
|
||||
except requests.exceptions.Timeout:
|
||||
debug_log(f"call_llm_streaming: timeout after {timeout_sec}s", "llm")
|
||||
return None
|
||||
except Exception as e:
|
||||
debug_log(f"call_llm_streaming: request failed — {e}", "llm")
|
||||
return None
|
||||
|
||||
|
||||
def extract_text_from_response(data: Dict[str, Any]) -> Optional[str]:
|
||||
"""Extract text from LLM response - supports multiple response formats."""
|
||||
# Preferred: Ollama chat non-stream format
|
||||
@@ -251,6 +459,29 @@ def chat_with_messages(
|
||||
|
||||
Returns the parsed JSON response dict on success, or None on error/timeout.
|
||||
"""
|
||||
if is_openai_compatible(base_url):
|
||||
opts = extra_options if isinstance(extra_options, dict) else {}
|
||||
body = openai_chat_body(
|
||||
chat_model, messages, stream=False, thinking=thinking,
|
||||
max_tokens=opts.get("num_predict"), temperature=opts.get("temperature"), tools=tools,
|
||||
)
|
||||
caller = _caller_name()
|
||||
try:
|
||||
return _openai_chat(base_url, body, timeout_sec, caller)
|
||||
except requests.exceptions.Timeout:
|
||||
print(" ⏱️ LLM request timed out", flush=True)
|
||||
except requests.exceptions.ConnectionError as e:
|
||||
print(f" ❌ LLM connection error: {e}", flush=True)
|
||||
except requests.exceptions.HTTPError as e:
|
||||
if e.response is not None and e.response.status_code == 400 and tools:
|
||||
raise ToolsNotSupportedError(
|
||||
f"Model {chat_model!r} returned HTTP 400 — native tools API not supported"
|
||||
)
|
||||
print(f" ❌ LLM HTTP error: {e}", flush=True)
|
||||
except Exception as e:
|
||||
print(f" ❌ LLM error: {e}", flush=True)
|
||||
return None
|
||||
|
||||
# Main agentic chat uses 8192 so the system prompt (tool list + protocol
|
||||
# guidance + memory context) doesn't overflow and force ollama to truncate
|
||||
# — which previously dropped the tool schema on smaller models like
|
||||
|
||||
@@ -2,7 +2,31 @@ from __future__ import annotations
|
||||
import requests
|
||||
|
||||
|
||||
def _openai_embedding(text: str, base_url: str, model: str, timeout_sec: float) -> list[float] | None:
|
||||
"""``/embeddings`` of an OpenAI-compatible server. A server without an
|
||||
embedding model (Strata serves the chat model only) answers an error, and
|
||||
the caller falls back to text search."""
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{base_url.rstrip('/')}/embeddings",
|
||||
json={"model": model, "input": text},
|
||||
timeout=timeout_sec,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json().get("data") or []
|
||||
vec = data[0].get("embedding") if data and isinstance(data[0], dict) else None
|
||||
if isinstance(vec, list):
|
||||
return [float(x) for x in vec]
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def get_embedding(text: str, base_url: str, model: str, timeout_sec: float = 15.0) -> list[float] | None:
|
||||
from ..llm import is_openai_compatible
|
||||
|
||||
if is_openai_compatible(base_url):
|
||||
return _openai_embedding(text, base_url, model, timeout_sec)
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{base_url.rstrip('/')}/api/embeddings",
|
||||
|
||||
@@ -77,7 +77,7 @@ Design principles enforced by the engine:
|
||||
6. Agentic Messages Loop with Dynamic Context
|
||||
- For each turn of the loop (max `agentic_max_turns` turns, default 8):
|
||||
- Update first system message with fresh time/location context
|
||||
- Send messages to LLM — try native tool calling first (Ollama `tools` API parameter)
|
||||
- Send messages to LLM — try native tool calling first (`tools` API parameter; Ollama `/api/chat`, or `/chat/completions` when `ollama_base_url` ends in `/v1`, see the transport note in `docs/llm_contexts.md`)
|
||||
- If the model returns HTTP 400 (native tools API not supported), automatically fall back
|
||||
to text-based tool calling for the rest of the session:
|
||||
- Rebuild system message to inject tool descriptions and markdown fence instructions
|
||||
@@ -140,7 +140,7 @@ Design principles enforced by the engine:
|
||||
- LLM returns empty response with no tool calls for multiple turns
|
||||
|
||||
Tool protocol details:
|
||||
- Native tool calling (default): Tools are passed to Ollama via the `tools` API parameter in OpenAI-compatible JSON schema format; the LLM requests tools via the standard `tool_calls` field
|
||||
- Native tool calling (default): Tools are passed to the LLM server (Ollama, or an OpenAI-compatible server such as Strata) via the `tools` API parameter in OpenAI-compatible JSON schema format; the LLM requests tools via the standard `tool_calls` field
|
||||
- Text-based fallback (automatic): If the model returns HTTP 400, the engine switches to injecting tool descriptions as plain text in the system message and parsing `` ```tool_call ``` `` markdown fences from the model's content field
|
||||
- Fallback is detected once per session (first HTTP 400 response) and persists for the rest of the conversation
|
||||
- Internal reasoning uses the `thinking` field (not shown to user)
|
||||
|
||||
@@ -38,7 +38,7 @@ Regardless of strategy, these tools are **always** included:
|
||||
|
||||
### Embedding Strategy
|
||||
|
||||
1. Embed the user query using `get_embedding()` (calls Ollama `/api/embeddings` with the configured embed model).
|
||||
1. Embed the user query using `get_embedding()` (calls Ollama `/api/embeddings` with the configured embed model, or `/embeddings` when `ollama_base_url` ends in `/v1`; no embedding means the strategy falls back as below).
|
||||
2. For each tool (excluding always-included), build a summary string from the tool name (camelCase split) and description, then embed it.
|
||||
3. Compute cosine similarity between the query embedding and each tool embedding.
|
||||
4. Select tools using a **relative threshold**: keep tools whose similarity >= `top_score * _RELATIVE_THRESHOLD` (0.97 — nomic-embed-text has a high baseline similarity, so a loose threshold lets the entire catalogue through).
|
||||
|
||||
Reference in New Issue
Block a user