perf(brain): pin chat model per-request, unload embeddings; default qwen2.5:3b
Replace the blunt global OLLAMA_KEEP_ALIVE=-1 (which kept every model, including nomic-embed, resident in VRAM forever) with per-request residency: - llm.py: all three /api/chat payloads send keep_alive=30m so the actively used chat model stays resident and voice turns never pay a cold reload. - embeddings.py: /api/embeddings sends keep_alive=0 so nomic-embed unloads right after each call instead of squatting in VRAM next to the chat model. - docker-compose.yml: drop the global OLLAMA_KEEP_ALIVE=-1; document the per-request scheme on the ollama service. Switch the default chat model qwen3:8b -> qwen2.5:3b. Verified live on the RTX 5050 (8GB): - ollama ps: qwen2.5:3b 2.4GB, 100% GPU (8B was 92% GPU / 8% CPU), UNTIL ~30m (the 30m pin, not "Forever"); nomic-embed absent after several enriched turns. - nvidia-smi: ~3.2GB VRAM used total (qwen 2.4GB + whisper 0.7GB) vs ~6.6GB. - Korean /text turns: warm 1.7-4s (cold first load ~52s), vs ~5-7s on 8B; time/weather/places tool calls fire and reply in Korean. Known limitation: qwen2.5:3b can occasionally leak a trailing CJK phrase on free-form chit-chat (factual/tool replies stay clean).
This commit is contained in:
@@ -17,11 +17,12 @@ services:
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
# Keep the chat + embed models resident so voice turns never pay a cold
|
||||
# reload. Default keep_alive is 5 min, so every post-idle turn took
|
||||
# ~30-60s while Qwen3 8B reloaded into the GPU. -1 = never unload.
|
||||
OLLAMA_KEEP_ALIVE: "-1"
|
||||
# Model residency is controlled per-request, not globally. The brain pins
|
||||
# the chat model with keep_alive=30m (src/jarvis/llm.py) so voice turns
|
||||
# never pay a cold reload, while embeddings pass keep_alive=0
|
||||
# (src/jarvis/memory/embeddings.py) so nomic-embed unloads right after use.
|
||||
# A global OLLAMA_KEEP_ALIVE=-1 was removed because it also kept the embed
|
||||
# model resident forever, wasting VRAM next to the chat model.
|
||||
volumes:
|
||||
- ollama_models:/root/.ollama
|
||||
# GPU: needs nvidia-container-toolkit on the host (CDI). Verified on the
|
||||
@@ -37,7 +38,7 @@ services:
|
||||
restart: "no"
|
||||
environment:
|
||||
OLLAMA_HOST: http://ollama:11434
|
||||
CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen3:8b}
|
||||
CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen2.5:3b}
|
||||
EMBED_MODEL: ${OLLAMA_EMBED_MODEL:-nomic-embed-text}
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
@@ -59,7 +60,7 @@ services:
|
||||
environment:
|
||||
# Point the brain at the ollama service and the bot at the in-container bridge.
|
||||
OLLAMA_BASE_URL: http://ollama:11434
|
||||
OLLAMA_CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen3:8b}
|
||||
OLLAMA_CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen2.5:3b}
|
||||
OLLAMA_EMBED_MODEL: ${OLLAMA_EMBED_MODEL:-nomic-embed-text}
|
||||
WHISPER_MODEL: ${WHISPER_MODEL:-small}
|
||||
WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda}
|
||||
|
||||
Reference in New Issue
Block a user