perf(memory): keep embed model warm across turns (keep_alive 0 -> 5m)

Empirical A/B/C measurement against the live RTX 5050 Ollama stack
(qwen2.5:3b + nomic-embed-text) showed keep_alive=0 unloads the embed
model ~2s after every call, so each turn after a brief idle gap pays a
cold reload. VRAM is not the constraint (~4.4-4.7 GB free with both
models resident) and keep_alive=0 never evicted the chat model, so CPU
embedding (num_gpu=0) gave no benefit. A short positive keep_alive is
the fastest of the three: it keeps the ~0.3 GB embed model resident
across consecutive turns at negligible VRAM cost.

Add tests/test_embeddings.py covering the warm-across-turns behaviour.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-12 23:45:16 +09:00
parent 67d0ae711c
commit 989a4f3e98
2 changed files with 60 additions and 5 deletions

View File

@@ -6,11 +6,14 @@ def get_embedding(text: str, base_url: str, model: str, timeout_sec: float = 15.
try:
resp = requests.post(
f"{base_url.rstrip('/')}/api/embeddings",
# keep_alive=0 unloads the embedding model right after the call so
# it does not sit resident in VRAM alongside the chat model. The
# chat model is pinned separately (llm.py keep_alive=30m); only the
# actively-used chat model should stay loaded.
json={"model": model, "prompt": text, "keep_alive": 0},
# Short positive keep_alive keeps the embed model warm across the
# consecutive turns of an active conversation. With keep_alive=0
# Ollama unloads it ~2s after every call, so each turn after a brief
# idle gap pays a cold reload of the embed model. The embed model is
# tiny (~0.3 GB) and coexists in VRAM with the chat model (pinned at
# keep_alive=30m in llm.py) with ample headroom, so holding it for a
# few minutes is effectively free and removes the per-turn reload.
json={"model": model, "prompt": text, "keep_alive": "5m"},
timeout=timeout_sec,
)
resp.raise_for_status()