perf(memory): keep embed model warm across turns (keep_alive 0 -> 5m)
Empirical A/B/C measurement against the live RTX 5050 Ollama stack (qwen2.5:3b + nomic-embed-text) showed keep_alive=0 unloads the embed model ~2s after every call, so each turn after a brief idle gap pays a cold reload. VRAM is not the constraint (~4.4-4.7 GB free with both models resident) and keep_alive=0 never evicted the chat model, so CPU embedding (num_gpu=0) gave no benefit. A short positive keep_alive is the fastest of the three: it keeps the ~0.3 GB embed model resident across consecutive turns at negligible VRAM cost. Add tests/test_embeddings.py covering the warm-across-turns behaviour. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -6,11 +6,14 @@ def get_embedding(text: str, base_url: str, model: str, timeout_sec: float = 15.
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{base_url.rstrip('/')}/api/embeddings",
|
||||
# keep_alive=0 unloads the embedding model right after the call so
|
||||
# it does not sit resident in VRAM alongside the chat model. The
|
||||
# chat model is pinned separately (llm.py keep_alive=30m); only the
|
||||
# actively-used chat model should stay loaded.
|
||||
json={"model": model, "prompt": text, "keep_alive": 0},
|
||||
# Short positive keep_alive keeps the embed model warm across the
|
||||
# consecutive turns of an active conversation. With keep_alive=0
|
||||
# Ollama unloads it ~2s after every call, so each turn after a brief
|
||||
# idle gap pays a cold reload of the embed model. The embed model is
|
||||
# tiny (~0.3 GB) and coexists in VRAM with the chat model (pinned at
|
||||
# keep_alive=30m in llm.py) with ample headroom, so holding it for a
|
||||
# few minutes is effectively free and removes the per-turn reload.
|
||||
json={"model": model, "prompt": text, "keep_alive": "5m"},
|
||||
timeout=timeout_sec,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
Reference in New Issue
Block a user