perf: cap chat output tokens via ollama_num_predict to bound reply latency
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only exposes the worst case where the chat model rambles or loops. Add an ollama_num_predict config (default 512, 0 disables) wired into the reply loop's chat call on both the native- and text-tool paths. The 512-token headroom stays well above this app's short tool-call JSON, so capping never truncates a tool call. This keeps the user's quality model instead of downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -4,6 +4,7 @@
|
||||
"ollama_base_url": "${OLLAMA_BASE_URL}",
|
||||
"ollama_embed_model": "${OLLAMA_EMBED_MODEL}",
|
||||
"ollama_chat_model": "${OLLAMA_CHAT_MODEL}",
|
||||
"ollama_num_predict": "${OLLAMA_NUM_PREDICT}",
|
||||
"intent_judge_model": "${OLLAMA_INTENT_MODEL}",
|
||||
"tts_enabled": true,
|
||||
"tts_engine": "${TTS_ENGINE}",
|
||||
|
||||
Reference in New Issue
Block a user