docs: reflect the separate auxiliary intent/router model in llm_contexts + README
Some checks failed
Release / semantic-release (push) Successful in 18s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m18s
Release / build-linux (push) Failing after 7m6s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Some checks failed
Release / semantic-release (push) Successful in 18s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m18s
Release / build-linux (push) Failing after 7m6s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Follow-up to the OLLAMA_INTENT_MODEL split: document that the Docker stack runs intent judging / tool routing / extraction on a small qwen2.5:3b (pulled by ollama-init) kept separate from the big chat answer model, and that setting OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL folds them back onto one resident model. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -87,7 +87,7 @@ COMPOSE_FILE=docker-compose.yml;docker-compose.gpu-windows.yml
|
|||||||
> Linux와 Windows는 GPU를 컨테이너에 넣는 방식이 달라서 override 파일이 갈립니다. Linux는 CDI(`devices: nvidia.com/gpu=all`), Windows(Docker Desktop)는 Compose의 `deploy.resources.reservations.devices`(`driver: nvidia`)를 씁니다. 호스트 사전 준비는 아래 "GPU 가속" 절 참고.
|
> Linux와 Windows는 GPU를 컨테이너에 넣는 방식이 달라서 override 파일이 갈립니다. Linux는 CDI(`devices: nvidia.com/gpu=all`), Windows(Docker Desktop)는 Compose의 `deploy.resources.reservations.devices`(`driver: nvidia`)를 씁니다. 호스트 사전 준비는 아래 "GPU 가속" 절 참고.
|
||||||
|
|
||||||
`docker compose up` 한 번이면 자동으로:
|
`docker compose up` 한 번이면 자동으로:
|
||||||
- Ollama 서버가 뜨고, `ollama-init`이 채팅/임베딩 모델을 **자동 pull**
|
- Ollama 서버가 뜨고, `ollama-init`이 채팅/보조(의도·라우팅)/임베딩 모델을 **자동 pull** (보조 모델 `OLLAMA_INTENT_MODEL`은 기본 `qwen2.5:3b`로, 큰 채팅 모델은 답변에만 쓰고 내부 분류 호출은 이 작은 모델이 처리)
|
||||||
- VNC+XFCE 데스크톱 + 크롬 + Python 브릿지가 기동
|
- VNC+XFCE 데스크톱 + 크롬 + Python 브릿지가 기동
|
||||||
- Whisper STT 모델 / Piper TTS 음성 자동 다운로드(볼륨에 캐시)
|
- Whisper STT 모델 / Piper TTS 음성 자동 다운로드(볼륨에 캐시)
|
||||||
|
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
|||||||
|
|
||||||
- **File**: [src/jarvis/listening/intent_judge.py](src/jarvis/listening/intent_judge.py) — `IntentJudge.evaluate()`.
|
- **File**: [src/jarvis/listening/intent_judge.py](src/jarvis/listening/intent_judge.py) — `IntentJudge.evaluate()`.
|
||||||
- **Trigger**: on a speech segment *only if* there is an engagement signal (wake word detected, hot-window active, or TTS playing). Pure ambient speech skips it.
|
- **Trigger**: on a speech segment *only if* there is an engagement signal (wake word detected, hot-window active, or TTS playing). Pure ambient speech skips it.
|
||||||
- **Model / gating**: `cfg.intent_judge_model` (default `gemma4:e2b`, ~2B). Falls back to text-based wake detection if Ollama is unavailable.
|
- **Model / gating**: `cfg.intent_judge_model`. Code-level default `gemma4:e2b` (~2B); the **Docker stack** renders it from `OLLAMA_INTENT_MODEL` (default `qwen2.5:3b`, pulled by `ollama-init`), kept deliberately **separate from `ollama_chat_model`** so this judge and the tool router (#3, #7) run on a small fast model while the big chat model is reserved for the spoken answer. Setting `OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL` folds them back onto one resident model. Falls back to text-based wake detection if Ollama is unavailable.
|
||||||
- **Inputs**:
|
- **Inputs**:
|
||||||
- Rolling transcript buffer (last 120s, with timestamps)
|
- Rolling transcript buffer (last 120s, with timestamps)
|
||||||
- Wake-word timestamp (if any), normalised aliases
|
- Wake-word timestamp (if any), normalised aliases
|
||||||
@@ -246,7 +246,7 @@ user input
|
|||||||
3. Pre-warm the intent-judge model before TTS finishes.
|
3. Pre-warm the intent-judge model before TTS finishes.
|
||||||
4. Cache tool-router (#7) output by query hash.
|
4. Cache tool-router (#7) output by query hash.
|
||||||
5. Give each digest its own timeout budget rather than sharing `llm_digest_timeout_sec` (today a slow memory digest can starve the max-turn digest).
|
5. Give each digest its own timeout budget rather than sharing `llm_digest_timeout_sec` (today a slow memory digest can starve the max-turn digest).
|
||||||
6. Consider single-model deployments: router+planner prefer `intent_judge_model`; loading a second model hurts cold-start latency on small hardware.
|
6. Two-model vs single-model tradeoff: the Docker default keeps a **separate** small `intent_judge_model` (`OLLAMA_INTENT_MODEL=qwen2.5:3b`) so routing/judging/extraction don't pay the big chat model's per-call cost — the main win once the GPU holds both models resident. On VRAM-constrained hardware, fold them onto one model by setting `OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL` (saves a resident model at the cost of slower routing when the chat model is large).
|
||||||
7. Narrow `llm_thinking_enabled` to router/planner only, not every context.
|
7. Narrow `llm_thinking_enabled` to router/planner only, not every context.
|
||||||
8. Reduce `intent_judge_timeout_sec` (15s) or race it against text-based wake detection to avoid blocking the audio loop.
|
8. Reduce `intent_judge_timeout_sec` (15s) or race it against text-based wake detection to avoid blocking the audio loop.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user