Empirical A/B/C measurement against the live RTX 5050 Ollama stack
(qwen2.5:3b + nomic-embed-text) showed keep_alive=0 unloads the embed
model ~2s after every call, so each turn after a brief idle gap pays a
cold reload. VRAM is not the constraint (~4.4-4.7 GB free with both
models resident) and keep_alive=0 never evicted the chat model, so CPU
embedding (num_gpu=0) gave no benefit. A short positive keep_alive is
the fastest of the three: it keeps the ~0.3 GB embed model resident
across consecutive turns at negligible VRAM cost.
Add tests/test_embeddings.py covering the warm-across-turns behaviour.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the blunt global OLLAMA_KEEP_ALIVE=-1 (which kept every model,
including nomic-embed, resident in VRAM forever) with per-request residency:
- llm.py: all three /api/chat payloads send keep_alive=30m so the actively
used chat model stays resident and voice turns never pay a cold reload.
- embeddings.py: /api/embeddings sends keep_alive=0 so nomic-embed unloads
right after each call instead of squatting in VRAM next to the chat model.
- docker-compose.yml: drop the global OLLAMA_KEEP_ALIVE=-1; document the
per-request scheme on the ollama service.
Switch the default chat model qwen3:8b -> qwen2.5:3b. Verified live on the
RTX 5050 (8GB):
- ollama ps: qwen2.5:3b 2.4GB, 100% GPU (8B was 92% GPU / 8% CPU), UNTIL ~30m
(the 30m pin, not "Forever"); nomic-embed absent after several enriched turns.
- nvidia-smi: ~3.2GB VRAM used total (qwen 2.4GB + whisper 0.7GB) vs ~6.6GB.
- Korean /text turns: warm 1.7-4s (cold first load ~52s), vs ~5-7s on 8B;
time/weather/places tool calls fire and reply in Korean.
Known limitation: qwen2.5:3b can occasionally leak a trailing CJK phrase on
free-form chit-chat (factual/tool replies stay clean).
