989a4f3e98
Empirical A/B/C measurement against the live RTX 5050 Ollama stack (qwen2.5:3b + nomic-embed-text) showed keep_alive=0 unloads the embed model ~2s after every call, so each turn after a brief idle gap pays a cold reload. VRAM is not the constraint (~4.4-4.7 GB free with both models resident) and keep_alive=0 never evicted the chat model, so CPU embedding (num_gpu=0) gave no benefit. A short positive keep_alive is the fastest of the three: it keeps the ~0.3 GB embed model resident across consecutive turns at negligible VRAM cost. Add tests/test_embeddings.py covering the warm-across-turns behaviour. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
27 lines
1.1 KiB
Python
27 lines
1.1 KiB
Python
from __future__ import annotations
|
|
import requests
|
|
|
|
|
|
def get_embedding(text: str, base_url: str, model: str, timeout_sec: float = 15.0) -> list[float] | None:
|
|
try:
|
|
resp = requests.post(
|
|
f"{base_url.rstrip('/')}/api/embeddings",
|
|
# Short positive keep_alive keeps the embed model warm across the
|
|
# consecutive turns of an active conversation. With keep_alive=0
|
|
# Ollama unloads it ~2s after every call, so each turn after a brief
|
|
# idle gap pays a cold reload of the embed model. The embed model is
|
|
# tiny (~0.3 GB) and coexists in VRAM with the chat model (pinned at
|
|
# keep_alive=30m in llm.py) with ample headroom, so holding it for a
|
|
# few minutes is effectively free and removes the per-turn reload.
|
|
json={"model": model, "prompt": text, "keep_alive": "5m"},
|
|
timeout=timeout_sec,
|
|
)
|
|
resp.raise_for_status()
|
|
data = resp.json()
|
|
vec = data.get("embedding")
|
|
if isinstance(vec, list):
|
|
return [float(x) for x in vec]
|
|
except Exception:
|
|
return None
|
|
return None
|