perf(brain): pin chat model per-request, unload embeddings; default qwen2.5:3b

Replace the blunt global OLLAMA_KEEP_ALIVE=-1 (which kept every model,
including nomic-embed, resident in VRAM forever) with per-request residency:

- llm.py: all three /api/chat payloads send keep_alive=30m so the actively
  used chat model stays resident and voice turns never pay a cold reload.
- embeddings.py: /api/embeddings sends keep_alive=0 so nomic-embed unloads
  right after each call instead of squatting in VRAM next to the chat model.
- docker-compose.yml: drop the global OLLAMA_KEEP_ALIVE=-1; document the
  per-request scheme on the ollama service.

Switch the default chat model qwen3:8b -> qwen2.5:3b. Verified live on the
RTX 5050 (8GB):
- ollama ps: qwen2.5:3b 2.4GB, 100% GPU (8B was 92% GPU / 8% CPU), UNTIL ~30m
  (the 30m pin, not "Forever"); nomic-embed absent after several enriched turns.
- nvidia-smi: ~3.2GB VRAM used total (qwen 2.4GB + whisper 0.7GB) vs ~6.6GB.
- Korean /text turns: warm 1.7-4s (cold first load ~52s), vs ~5-7s on 8B;
  time/weather/places tool calls fire and reply in Korean.

Known limitation: qwen2.5:3b can occasionally leak a trailing CJK phrase on
free-form chit-chat (factual/tool replies stay clean).
This commit is contained in:
javis-bot
2026-06-12 20:36:19 +09:00
parent 7792be254a
commit b91c05a355
4 changed files with 29 additions and 11 deletions

View File

@@ -17,11 +17,12 @@ services:
ollama:
image: ollama/ollama:latest
restart: unless-stopped
environment:
# Keep the chat + embed models resident so voice turns never pay a cold
# reload. Default keep_alive is 5 min, so every post-idle turn took
# ~30-60s while Qwen3 8B reloaded into the GPU. -1 = never unload.
OLLAMA_KEEP_ALIVE: "-1"
# Model residency is controlled per-request, not globally. The brain pins
# the chat model with keep_alive=30m (src/jarvis/llm.py) so voice turns
# never pay a cold reload, while embeddings pass keep_alive=0
# (src/jarvis/memory/embeddings.py) so nomic-embed unloads right after use.
# A global OLLAMA_KEEP_ALIVE=-1 was removed because it also kept the embed
# model resident forever, wasting VRAM next to the chat model.
volumes:
- ollama_models:/root/.ollama
# GPU: needs nvidia-container-toolkit on the host (CDI). Verified on the
@@ -37,7 +38,7 @@ services:
restart: "no"
environment:
OLLAMA_HOST: http://ollama:11434
CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen3:8b}
CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen2.5:3b}
EMBED_MODEL: ${OLLAMA_EMBED_MODEL:-nomic-embed-text}
entrypoint: ["/bin/sh", "-c"]
command:
@@ -59,7 +60,7 @@ services:
environment:
# Point the brain at the ollama service and the bot at the in-container bridge.
OLLAMA_BASE_URL: http://ollama:11434
OLLAMA_CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen3:8b}
OLLAMA_CHAT_MODEL: ${OLLAMA_CHAT_MODEL:-qwen2.5:3b}
OLLAMA_EMBED_MODEL: ${OLLAMA_EMBED_MODEL:-nomic-embed-text}
WHISPER_MODEL: ${WHISPER_MODEL:-small}
WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda}