javis-bot 989a4f3e98 perf(memory): keep embed model warm across turns (keep_alive 0 -> 5m) ...

Empirical A/B/C measurement against the live RTX 5050 Ollama stack
(qwen2.5:3b + nomic-embed-text) showed keep_alive=0 unloads the embed
model ~2s after every call, so each turn after a brief idle gap pays a
cold reload. VRAM is not the constraint (~4.4-4.7 GB free with both
models resident) and keep_alive=0 never evicted the chat model, so CPU
embedding (num_gpu=0) gave no benefit. A short positive keep_alive is
the fastest of the three: it keeps the ~0.3 GB embed model resident
across consecutive turns at negligible VRAM cost.

Add tests/test_embeddings.py covering the warm-across-turns behaviour.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-06-12 23:45:16 +09:00

2.2 KiB

Raw Blame History

View Raw