b52ffd2b18
Some checks failed
Release / semantic-release (push) Successful in 19s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled
Intent judging, tool routing and arg extraction are classification/JSON calls, not the spoken answer, yet the stack aliased OLLAMA_INTENT_MODEL back to the big chat model — so each command paid the big model's cost 2-3 times for routing before the reply even ran. With the GPU on, that round-trip stacking is the main remaining per-turn latency. Default OLLAMA_INTENT_MODEL to qwen2.5:3b (the project's reference small model, clean Korean on classification) and have ollama-init pull it. The reply engine already routes these calls through intent_judge_model, so answer quality is untouched; set OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL to fold back onto one resident model. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
63 lines
2.6 KiB
Python
63 lines
2.6 KiB
Python
"""The docker deployment must run auxiliary calls on a small model.
|
|
|
|
Latency win: intent judging, tool routing and arg extraction are
|
|
classification/JSON calls, not the spoken answer. Running them on a small fast
|
|
model means the big chat model only runs once per command (for the answer),
|
|
instead of 2-3 times per command for routing/extraction too.
|
|
|
|
The wiring is: docker/jarvis-config.template.json renders `intent_judge_model`
|
|
from `${OLLAMA_INTENT_MODEL}`, and the reply engine's resolver falls through
|
|
`tool_router_model -> intent_judge_model -> ollama_chat_model`. The template
|
|
sets no `tool_router_model`, so the auxiliary model is whatever
|
|
`OLLAMA_INTENT_MODEL` renders to. These tests pin that behaviour end to end.
|
|
"""
|
|
|
|
import json
|
|
import string
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from jarvis.reply.engine import resolve_tool_router_model
|
|
|
|
TEMPLATE = Path(__file__).resolve().parent.parent / "docker" / "jarvis-config.template.json"
|
|
|
|
|
|
def _render(**env) -> dict:
|
|
raw = TEMPLATE.read_text(encoding="utf-8")
|
|
return json.loads(string.Template(raw).safe_substitute(**env))
|
|
|
|
|
|
class _Cfg:
|
|
"""cfg stand-in carrying only the fields the resolver reads. The template
|
|
does not render `tool_router_model`, so it stays empty here too."""
|
|
|
|
def __init__(self, rendered: dict):
|
|
self.tool_router_model = rendered.get("tool_router_model", "") or ""
|
|
self.intent_judge_model = rendered.get("intent_judge_model", "") or ""
|
|
self.ollama_chat_model = rendered.get("ollama_chat_model", "") or ""
|
|
|
|
|
|
def test_template_renders_separate_intent_model():
|
|
cfg = _render(OLLAMA_CHAT_MODEL="qwen3:8b", OLLAMA_INTENT_MODEL="qwen2.5:3b")
|
|
assert cfg["ollama_chat_model"] == "qwen3:8b"
|
|
assert cfg["intent_judge_model"] == "qwen2.5:3b"
|
|
assert cfg["intent_judge_model"] != cfg["ollama_chat_model"]
|
|
|
|
|
|
@pytest.mark.unit
|
|
def test_aux_calls_route_to_small_model_not_chat_model():
|
|
"""The whole point: with a big chat model and a small intent model, tool
|
|
routing must resolve to the small model, leaving the big model for answers."""
|
|
cfg = _Cfg(_render(OLLAMA_CHAT_MODEL="qwen3:8b", OLLAMA_INTENT_MODEL="qwen2.5:3b"))
|
|
assert resolve_tool_router_model(cfg) == "qwen2.5:3b"
|
|
|
|
|
|
@pytest.mark.unit
|
|
def test_folding_intent_onto_chat_model_keeps_one_model():
|
|
"""Setting OLLAMA_INTENT_MODEL == OLLAMA_CHAT_MODEL folds everything back
|
|
onto a single resident model (the documented VRAM-saving opt-out)."""
|
|
cfg = _Cfg(_render(OLLAMA_CHAT_MODEL="qwen2.5:3b", OLLAMA_INTENT_MODEL="qwen2.5:3b"))
|
|
assert resolve_tool_router_model(cfg) == "qwen2.5:3b"
|
|
assert cfg.intent_judge_model == cfg.ollama_chat_model
|