perf: run auxiliary LLM calls on a small model, big model only for the answer
Some checks failed
Release / semantic-release (push) Successful in 19s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled
Some checks failed
Release / semantic-release (push) Successful in 19s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled
Intent judging, tool routing and arg extraction are classification/JSON calls, not the spoken answer, yet the stack aliased OLLAMA_INTENT_MODEL back to the big chat model — so each command paid the big model's cost 2-3 times for routing before the reply even ran. With the GPU on, that round-trip stacking is the main remaining per-turn latency. Default OLLAMA_INTENT_MODEL to qwen2.5:3b (the project's reference small model, clean Korean on classification) and have ollama-init pull it. The reply engine already routes these calls through intent_judge_model, so answer quality is untouched; set OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL to fold back onto one resident model. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
11
.env.example
11
.env.example
@@ -59,11 +59,12 @@ OLLAMA_BASE_URL=http://127.0.0.1:11434
|
||||
# free-form chit-chat. Swap back to qwen3:8b for the strongest tool-calling.
|
||||
OLLAMA_CHAT_MODEL=qwen2.5:3b
|
||||
# Model for the auxiliary small-model calls: intent judge, tool router, weather
|
||||
# place extraction, query decomposition. BLANK (default) reuses OLLAMA_CHAT_MODEL
|
||||
# so the stack runs on one already-warm model. The code's built-in default
|
||||
# (gemma4:e2b) is NOT pulled by this stack, so leaving this unset previously made
|
||||
# every router/extractor call silently fail. Only set this if you also pull the
|
||||
# model into Ollama.
|
||||
# place extraction, query decomposition. These are classification/JSON calls,
|
||||
# NOT the spoken answer, so a small fast model here cuts 2-3 big-model round
|
||||
# trips per command without touching answer quality. BLANK uses the stack
|
||||
# default qwen2.5:3b, which ollama-init pulls automatically. Set it equal to
|
||||
# OLLAMA_CHAT_MODEL to run everything on one resident model instead (saves VRAM
|
||||
# at the cost of slower routing when the chat model is large).
|
||||
OLLAMA_INTENT_MODEL=
|
||||
OLLAMA_EMBED_MODEL=nomic-embed-text
|
||||
WHISPER_MODEL=small
|
||||
|
||||
Reference in New Issue
Block a user