perf: unify Ollama num_ctx so a voice turn keeps one resident model

Ollama keeps a separate loaded model instance per (model, num_ctx). The
main agentic chat used num_ctx=8192 while the router/enrichment/digest
passes used 4096, so every voice turn forced at least one cold reload
(~3.4s) when switching context sizes — the dominant per-turn latency
(measured: resident chat call 0.27s vs cold 3.4s).

Introduce a single OLLAMA_NUM_CTX (default 8192, env-tunable for tight
VRAM) used by call_llm_direct, chat_with_messages, call_llm_streaming and
the planner, collapsing a turn to one resident instance.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-14 00:19:53 +09:00
parent d4e5e7f3f7
commit 2c38e7576d
2 changed files with 23 additions and 10 deletions

View File

@@ -40,7 +40,7 @@ import re
from typing import List, Optional, Sequence, Tuple
from ..debug import debug_log
from ..llm import call_llm_direct
from ..llm import call_llm_direct, OLLAMA_NUM_CTX
# Hard cap on plan length. Small models happily emit 10+ step plans that
@@ -441,7 +441,7 @@ def plan_query(
user_content=user_content,
timeout_sec=effective_timeout,
thinking=False,
num_ctx=8192,
num_ctx=OLLAMA_NUM_CTX,
)
except Exception as exc: # pragma: no cover — defensive
debug_log(f"planner: LLM call failed — {exc}", "planning")
@@ -716,7 +716,7 @@ def resolve_next_tool_call(
user_content=user_content,
timeout_sec=effective_timeout,
thinking=False,
num_ctx=8192,
num_ctx=OLLAMA_NUM_CTX,
)
except Exception as exc: # pragma: no cover — defensive
debug_log(f"planner.resolve_next_tool_call: LLM failed — {exc}", "planning")