docs: reflect the separate auxiliary intent/router model in llm_contexts + README
Some checks failed
Release / semantic-release (push) Successful in 18s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m18s
Release / build-linux (push) Failing after 7m6s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled

Follow-up to the OLLAMA_INTENT_MODEL split: document that the Docker stack runs
intent judging / tool routing / extraction on a small qwen2.5:3b (pulled by
ollama-init) kept separate from the big chat answer model, and that setting
OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL folds them back onto one resident model.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-23 17:38:58 +09:00
parent b52ffd2b18
commit 680f5a656a
2 changed files with 3 additions and 3 deletions

View File

@@ -26,7 +26,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- **File**: [src/jarvis/listening/intent_judge.py](src/jarvis/listening/intent_judge.py) — `IntentJudge.evaluate()`.
- **Trigger**: on a speech segment *only if* there is an engagement signal (wake word detected, hot-window active, or TTS playing). Pure ambient speech skips it.
- **Model / gating**: `cfg.intent_judge_model` (default `gemma4:e2b`, ~2B). Falls back to text-based wake detection if Ollama is unavailable.
- **Model / gating**: `cfg.intent_judge_model`. Code-level default `gemma4:e2b` (~2B); the **Docker stack** renders it from `OLLAMA_INTENT_MODEL` (default `qwen2.5:3b`, pulled by `ollama-init`), kept deliberately **separate from `ollama_chat_model`** so this judge and the tool router (#3, #7) run on a small fast model while the big chat model is reserved for the spoken answer. Setting `OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL` folds them back onto one resident model. Falls back to text-based wake detection if Ollama is unavailable.
- **Inputs**:
- Rolling transcript buffer (last 120s, with timestamps)
- Wake-word timestamp (if any), normalised aliases
@@ -246,7 +246,7 @@ user input
3. Pre-warm the intent-judge model before TTS finishes.
4. Cache tool-router (#7) output by query hash.
5. Give each digest its own timeout budget rather than sharing `llm_digest_timeout_sec` (today a slow memory digest can starve the max-turn digest).
6. Consider single-model deployments: router+planner prefer `intent_judge_model`; loading a second model hurts cold-start latency on small hardware.
6. Two-model vs single-model tradeoff: the Docker default keeps a **separate** small `intent_judge_model` (`OLLAMA_INTENT_MODEL=qwen2.5:3b`) so routing/judging/extraction don't pay the big chat model's per-call cost — the main win once the GPU holds both models resident. On VRAM-constrained hardware, fold them onto one model by setting `OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL` (saves a resident model at the cost of slower routing when the chat model is large).
7. Narrow `llm_thinking_enabled` to router/planner only, not every context.
8. Reduce `intent_judge_timeout_sec` (15s) or race it against text-based wake detection to avoid blocking the audio loop.