tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

perf: cap chat output tokens via ollama_num_predict to bound reply latency

Browse Source 
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only
exposes the worst case where the chat model rambles or loops. Add an
ollama_num_predict config (default 512, 0 disables) wired into the reply
loop's chat call on both the native- and text-tool paths. The 512-token
headroom stays well above this app's short tool-call JSON, so capping never
truncates a tool call. This keeps the user's quality model instead of
downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-23 15:33:45 +09:00
parent c189ce2e65
commit 5ee47827f3
7 changed files with 146 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/llm_contexts.md
View File

@@ -20,7 +20,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68))
- Tool results from prior turns (raw or digested — see #5)
- **Output**: OpenAI-style `{content, tool_calls, thinking}`. Consumed by the tool orchestrator and TTS pipeline. Natural-language content is delivered immediately; no post-turn evaluator runs.
- **Limits**: `num_ctx: 8192` (explicit). Timeout `llm_chat_timeout_sec` (45s). Auto-fallback from native to text tool-calls on HTTP 400 (`ToolsNotSupportedError`), sticky for the session. Risk: `fetch_web_page` truncates at 50,000 chars (~37k tokens) — mitigated for SMALL models by tool-result digest (#5) which compresses the payload before it enters the messages history. LARGE models receive the raw payload and may silently see a truncated context.
- **Limits**: `num_ctx: 8192` (explicit). Output `num_predict: cfg.ollama_num_predict` (default 512, `0`/negative disables) caps generated tokens per turn — a worst-case latency guard for short spoken answers; the headroom stays above tool-call JSON so it does not truncate tool calls (both native and text tool-call paths). Timeout `llm_chat_timeout_sec` (45s). Auto-fallback from native to text tool-calls on HTTP 400 (`ToolsNotSupportedError`), sticky for the session. Risk: `fetch_web_page` truncates at 50,000 chars (~37k tokens) — mitigated for SMALL models by tool-result digest (#5) which compresses the payload before it enters the messages history. LARGE models receive the raw payload and may silently see a truncated context.
## 2. Intent Judge