perf: run auxiliary LLM calls on a small model, big model only for the answer
Some checks failed
Release / semantic-release (push) Successful in 19s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled

Intent judging, tool routing and arg extraction are classification/JSON calls,
not the spoken answer, yet the stack aliased OLLAMA_INTENT_MODEL back to the big
chat model — so each command paid the big model's cost 2-3 times for routing
before the reply even ran. With the GPU on, that round-trip stacking is the main
remaining per-turn latency. Default OLLAMA_INTENT_MODEL to qwen2.5:3b (the
project's reference small model, clean Korean on classification) and have
ollama-init pull it. The reply engine already routes these calls through
intent_judge_model, so answer quality is untouched; set OLLAMA_INTENT_MODEL =
OLLAMA_CHAT_MODEL to fold back onto one resident model.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-23 17:35:40 +09:00
parent 140fc56f18
commit b52ffd2b18
4 changed files with 89 additions and 11 deletions

View File

@@ -59,11 +59,12 @@ OLLAMA_BASE_URL=http://127.0.0.1:11434
# free-form chit-chat. Swap back to qwen3:8b for the strongest tool-calling.
OLLAMA_CHAT_MODEL=qwen2.5:3b
# Model for the auxiliary small-model calls: intent judge, tool router, weather
# place extraction, query decomposition. BLANK (default) reuses OLLAMA_CHAT_MODEL
# so the stack runs on one already-warm model. The code's built-in default
# (gemma4:e2b) is NOT pulled by this stack, so leaving this unset previously made
# every router/extractor call silently fail. Only set this if you also pull the
# model into Ollama.
# place extraction, query decomposition. These are classification/JSON calls,
# NOT the spoken answer, so a small fast model here cuts 2-3 big-model round
# trips per command without touching answer quality. BLANK uses the stack
# default qwen2.5:3b, which ollama-init pulls automatically. Set it equal to
# OLLAMA_CHAT_MODEL to run everything on one resident model instead (saves VRAM
# at the cost of slower routing when the chat model is large).
OLLAMA_INTENT_MODEL=
OLLAMA_EMBED_MODEL=nomic-embed-text
WHISPER_MODEL=small