tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

perf: cap chat output tokens via ollama_num_predict to bound reply latency

Browse Source 
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only
exposes the worst case where the chat model rambles or loops. Add an
ollama_num_predict config (default 512, 0 disables) wired into the reply
loop's chat call on both the native- and text-tool paths. The 512-token
headroom stays well above this app's short tool-call JSON, so capping never
truncates a tool call. This keeps the user's quality model instead of
downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-23 15:33:45 +09:00
parent c189ce2e65
commit 5ee47827f3
7 changed files with 146 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docker/entrypoint.sh
View File

@@ -16,6 +16,9 @@ set -euo pipefail
# by default so everything runs on one resident model; override if you pull a
# dedicated small model.
: "${OLLAMA_INTENT_MODEL:=${OLLAMA_CHAT_MODEL}}"
# Cap chat-model output tokens per turn (worst-case latency guard). Spoken
# answers are 1-2 sentences; 512 is safe headroom above tool-call JSON. 0 = off.
: "${OLLAMA_NUM_PREDICT:=512}"
: "${OLLAMA_EMBED_MODEL:=nomic-embed-text}"
: "${WHISPER_MODEL:=small}"
: "${WHISPER_DEVICE:=cuda}"
@@ -32,7 +35,7 @@ set -euo pipefail
: "${XDG_RUNTIME_DIR:=/run/user/0}"
: "${PULSE_SERVER:=unix:${XDG_RUNTIME_DIR}/pulse/native}"
export VNC_RESOLUTION OLLAMA_BASE_URL OLLAMA_CHAT_MODEL OLLAMA_INTENT_MODEL OLLAMA_EMBED_MODEL \
export VNC_RESOLUTION OLLAMA_BASE_URL OLLAMA_CHAT_MODEL OLLAMA_NUM_PREDICT OLLAMA_INTENT_MODEL OLLAMA_EMBED_MODEL \
WHISPER_MODEL WHISPER_DEVICE WHISPER_COMPUTE_TYPE JARVIS_DB_PATH \
PIPER_VOICE PIPER_VOICE_DIR TTS_PIPER_MODEL_PATH BRIDGE_HOST BRIDGE_PORT \
XDG_RUNTIME_DIR PULSE_SERVER