docs: reflect the separate auxiliary intent/router model in llm_contexts + README
Some checks failed
Release / semantic-release (push) Successful in 18s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m18s
Release / build-linux (push) Failing after 7m6s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Some checks failed
Release / semantic-release (push) Successful in 18s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m18s
Release / build-linux (push) Failing after 7m6s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Follow-up to the OLLAMA_INTENT_MODEL split: document that the Docker stack runs intent judging / tool routing / extraction on a small qwen2.5:3b (pulled by ollama-init) kept separate from the big chat answer model, and that setting OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL folds them back onto one resident model. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -26,7 +26,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
|
||||
- **File**: [src/jarvis/listening/intent_judge.py](src/jarvis/listening/intent_judge.py) — `IntentJudge.evaluate()`.
|
||||
- **Trigger**: on a speech segment *only if* there is an engagement signal (wake word detected, hot-window active, or TTS playing). Pure ambient speech skips it.
|
||||
- **Model / gating**: `cfg.intent_judge_model` (default `gemma4:e2b`, ~2B). Falls back to text-based wake detection if Ollama is unavailable.
|
||||
- **Model / gating**: `cfg.intent_judge_model`. Code-level default `gemma4:e2b` (~2B); the **Docker stack** renders it from `OLLAMA_INTENT_MODEL` (default `qwen2.5:3b`, pulled by `ollama-init`), kept deliberately **separate from `ollama_chat_model`** so this judge and the tool router (#3, #7) run on a small fast model while the big chat model is reserved for the spoken answer. Setting `OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL` folds them back onto one resident model. Falls back to text-based wake detection if Ollama is unavailable.
|
||||
- **Inputs**:
|
||||
- Rolling transcript buffer (last 120s, with timestamps)
|
||||
- Wake-word timestamp (if any), normalised aliases
|
||||
@@ -246,7 +246,7 @@ user input
|
||||
3. Pre-warm the intent-judge model before TTS finishes.
|
||||
4. Cache tool-router (#7) output by query hash.
|
||||
5. Give each digest its own timeout budget rather than sharing `llm_digest_timeout_sec` (today a slow memory digest can starve the max-turn digest).
|
||||
6. Consider single-model deployments: router+planner prefer `intent_judge_model`; loading a second model hurts cold-start latency on small hardware.
|
||||
6. Two-model vs single-model tradeoff: the Docker default keeps a **separate** small `intent_judge_model` (`OLLAMA_INTENT_MODEL=qwen2.5:3b`) so routing/judging/extraction don't pay the big chat model's per-call cost — the main win once the GPU holds both models resident. On VRAM-constrained hardware, fold them onto one model by setting `OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL` (saves a resident model at the cost of slower routing when the chat model is large).
|
||||
7. Narrow `llm_thinking_enabled` to router/planner only, not every context.
|
||||
8. Reduce `intent_judge_timeout_sec` (15s) or race it against text-based wake detection to avoid blocking the audio loop.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user