perf: unify Ollama num_ctx so a voice turn keeps one resident model
Ollama keeps a separate loaded model instance per (model, num_ctx). The main agentic chat used num_ctx=8192 while the router/enrichment/digest passes used 4096, so every voice turn forced at least one cold reload (~3.4s) when switching context sizes — the dominant per-turn latency (measured: resident chat call 0.27s vs cold 3.4s). Introduce a single OLLAMA_NUM_CTX (default 8192, env-tunable for tight VRAM) used by call_llm_direct, chat_with_messages, call_llm_streaming and the planner, collapsing a turn to one resident instance. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -40,7 +40,7 @@ import re
|
||||
from typing import List, Optional, Sequence, Tuple
|
||||
|
||||
from ..debug import debug_log
|
||||
from ..llm import call_llm_direct
|
||||
from ..llm import call_llm_direct, OLLAMA_NUM_CTX
|
||||
|
||||
|
||||
# Hard cap on plan length. Small models happily emit 10+ step plans that
|
||||
@@ -441,7 +441,7 @@ def plan_query(
|
||||
user_content=user_content,
|
||||
timeout_sec=effective_timeout,
|
||||
thinking=False,
|
||||
num_ctx=8192,
|
||||
num_ctx=OLLAMA_NUM_CTX,
|
||||
)
|
||||
except Exception as exc: # pragma: no cover — defensive
|
||||
debug_log(f"planner: LLM call failed — {exc}", "planning")
|
||||
@@ -716,7 +716,7 @@ def resolve_next_tool_call(
|
||||
user_content=user_content,
|
||||
timeout_sec=effective_timeout,
|
||||
thinking=False,
|
||||
num_ctx=8192,
|
||||
num_ctx=OLLAMA_NUM_CTX,
|
||||
)
|
||||
except Exception as exc: # pragma: no cover — defensive
|
||||
debug_log(f"planner.resolve_next_tool_call: LLM failed — {exc}", "planning")
|
||||
|
||||
Reference in New Issue
Block a user