perf(brain): pin chat model per-request, unload embeddings; default qwen2.5:3b
Replace the blunt global OLLAMA_KEEP_ALIVE=-1 (which kept every model, including nomic-embed, resident in VRAM forever) with per-request residency: - llm.py: all three /api/chat payloads send keep_alive=30m so the actively used chat model stays resident and voice turns never pay a cold reload. - embeddings.py: /api/embeddings sends keep_alive=0 so nomic-embed unloads right after each call instead of squatting in VRAM next to the chat model. - docker-compose.yml: drop the global OLLAMA_KEEP_ALIVE=-1; document the per-request scheme on the ollama service. Switch the default chat model qwen3:8b -> qwen2.5:3b. Verified live on the RTX 5050 (8GB): - ollama ps: qwen2.5:3b 2.4GB, 100% GPU (8B was 92% GPU / 8% CPU), UNTIL ~30m (the 30m pin, not "Forever"); nomic-embed absent after several enriched turns. - nvidia-smi: ~3.2GB VRAM used total (qwen 2.4GB + whisper 0.7GB) vs ~6.6GB. - Korean /text turns: warm 1.7-4s (cold first load ~52s), vs ~5-7s on 8B; time/weather/places tool calls fire and reply in Korean. Known limitation: qwen2.5:3b can occasionally leak a trailing CJK phrase on free-form chit-chat (factual/tool replies stay clean).
This commit is contained in:
@@ -47,8 +47,11 @@ MELO_SPEED=1.5
|
||||
# ---------------------------------------------------------------------------
|
||||
# In docker-compose this is overridden to http://ollama:11434 automatically.
|
||||
OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||
# qwen3:8b — best 8GB-VRAM pick: strongest tool-calling, ~5GB Q4, fits the RTX 5050.
|
||||
OLLAMA_CHAT_MODEL=qwen3:8b
|
||||
# qwen2.5:3b — small non-reasoning instruct model. ~2.4GB, runs 100% on the GPU
|
||||
# (the 8B offloads ~8% to CPU), warm voice turns ~2-4s vs ~5-7s on 8B. Clean
|
||||
# Korean on factual/tool replies; can occasionally leak a trailing CJK phrase on
|
||||
# free-form chit-chat. Swap back to qwen3:8b for the strongest tool-calling.
|
||||
OLLAMA_CHAT_MODEL=qwen2.5:3b
|
||||
OLLAMA_EMBED_MODEL=nomic-embed-text
|
||||
WHISPER_MODEL=small
|
||||
|
||||
|
||||
Reference in New Issue
Block a user