tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

perf: cap chat output tokens via ollama_num_predict to bound reply latency

Browse Source 
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only
exposes the worst case where the chat model rambles or loops. Add an
ollama_num_predict config (default 512, 0 disables) wired into the reply
loop's chat call on both the native- and text-tool paths. The 512-token
headroom stays well above this app's short tool-call JSON, so capping never
truncates a tool call. This keeps the user's quality model instead of
downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-23 15:33:45 +09:00
parent c189ce2e65
commit 5ee47827f3
7 changed files with 146 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
src/jarvis/config.py
View File

@@ -85,6 +85,12 @@ class Settings:
llm_digest_timeout_sec: float
llm_embedding_timeout_sec: float
llm_profile_select_timeout_sec: float
# Upper bound on tokens the chat model may generate per reply turn. Spoken
# (TTS) answers are 1-2 sentences, so a cap bounds the worst-case latency of
# a model that occasionally rambles or loops without changing normal answers.
# The headroom (default 512) sits well above this app's short tool-call JSON,
# so capping never truncates a tool call. 0 (or negative) disables the cap.
ollama_num_predict: int
# Profiles & Behavior
active_profiles: list[str]
@@ -394,6 +400,9 @@ def get_default_config() -> Dict[str, Any]:
"llm_digest_timeout_sec": 8.0,
"llm_embedding_timeout_sec": 60.0,
"llm_profile_select_timeout_sec": 30.0,
# Cap on chat-model output tokens per turn (worst-case latency guard).
# 512 is safe headroom above short TTS answers and tool-call JSON; 0 off.
"ollama_num_predict": 512,
# Profiles & Behavior
"active_profiles": ["developer", "business", "life"],
@@ -763,6 +772,10 @@ def load_settings() -> Settings:
llm_digest_timeout_sec = float(merged.get("llm_digest_timeout_sec", 8.0))
llm_embedding_timeout_sec = float(merged.get("llm_embedding_timeout_sec", 60.0))
llm_profile_select_timeout_sec = float(merged.get("llm_profile_select_timeout_sec", 30.0))
try:
ollama_num_predict = int(merged.get("ollama_num_predict", 512))
except (TypeError, ValueError):
ollama_num_predict = 512
return Settings(
# Database & Storage
@@ -778,6 +791,7 @@ def load_settings() -> Settings:
llm_digest_timeout_sec=llm_digest_timeout_sec,
llm_embedding_timeout_sec=llm_embedding_timeout_sec,
llm_profile_select_timeout_sec=llm_profile_select_timeout_sec,
ollama_num_predict=ollama_num_predict,
# Profiles & Behavior
active_profiles=active_profiles,