tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Add Discord-native hybrid front-end for Jarvis (bot + bridge)
Some checks failed
Release / semantic-release (push) Successful in 59s
Details
tests / Unit tests (Linux, Python 3.11) (push) Successful in 13m45s
Details
Release / build-linux (push) Failing after 7m47s
Details
Release / build-windows (push) Has been cancelled
Details
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Details
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Details
Release / release-main (push) Has been cancelled
Details
Release / release-develop (push) Has been cancelled
Details

Browse Source 
Transform isair/jarvis into a Discord-controlled voice assistant running on
the Ubuntu VNC desktop, keeping the mature ~39k-line Python brain intact.

- bot/ (Node + bun, discord.js): /자비스 slash commands (ephemeral),
  voice channel join + voice receive/playback, pluggable VNC screen broadcast
  (selfbot live / noVNC / screenshot)
- bridge/ (Python, Flask): wraps jarvis STT + run_reply_engine + Piper TTS
  behind a thin localhost HTTP API
- .env.example, scripts/ (start_bridge/start_bot/dev), README rewrite,
  docs/language-comparison.md and docs/vnc-xfce-setup.md

Language decision: hybrid (Python brain + Node/bun Discord layer) because
Discord blocks bot video; native screen broadcast only works via a Node
selfbot library.
This commit is contained in:
javis-bot
2026-06-09 14:51:05 +09:00
parent a5bf8d1826
commit c4abf63f38
308 changed files with 94135 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
src/jarvis/reply/prompts/__init__.py Normal file
View File

@@ -0,0 +1,19 @@
"""
Prompt system for model-size-aware response generation.
This module provides model-size-specific prompt variations to improve
tool usage accuracy across different LLM sizes.
"""
from .model_variants import ModelSize, detect_model_size, get_system_prompts
from .system import PromptComponents, ASR_NOTE, INFERENCE_GUIDANCE, VOICE_STYLE
__all__ = [
"ModelSize",
"detect_model_size",
"get_system_prompts",
"PromptComponents",
"ASR_NOTE",
"INFERENCE_GUIDANCE",
"VOICE_STYLE",
]

244
src/jarvis/reply/prompts/model_variants.py Normal file
View File

@@ -0,0 +1,244 @@
"""
Model-size-specific prompt variations.
Small models (1b, 3b, 7b) need explicit guidance on when NOT to use tools,
while larger models can infer this from context.
"""
from enum import Enum
from typing import Optional
from .system import (
PromptComponents,
ASR_NOTE,
INFERENCE_GUIDANCE,
VOICE_STYLE,
)
class ModelSize(Enum):
"""Classification of model sizes for prompt selection."""
SMALL = "small" # 1b, 3b, 7b - needs explicit tool constraints
LARGE = "large" # 8b+ - can infer tool usage from context
# Model size patterns - models matching these are considered SMALL
_SMALL_MODEL_PATTERNS = (
":1b", ":3b", ":7b",
"-1b", "-3b", "-7b",
"_1b", "_3b", "_7b",
"gemma4", # Gemma 4 - always small regardless of tag
)
def detect_model_size(model_name: Optional[str]) -> ModelSize:
"""
Detect model size from model name.
Args:
model_name: Ollama model name (e.g., "gemma4", "gpt-oss:20b")
Returns:
ModelSize.SMALL for 1b/3b/7b models, ModelSize.LARGE otherwise
"""
if not model_name:
return ModelSize.LARGE # Default to large for safety
name_lower = model_name.lower()
for pattern in _SMALL_MODEL_PATTERNS:
if pattern in name_lower:
return ModelSize.SMALL
return ModelSize.LARGE
# =============================================================================
# Large Model Prompts
# =============================================================================
TOOL_INCENTIVES_LARGE = (
"Proactively use available tools to provide better, more accurate responses. "
"Prefer tools over guessing when you can get definitive, current, or personalized information. "
"Tools enhance your capabilities - use them confidently to deliver superior assistance. "
"Always try tools before asking the user for information you might already be able to get via them."
)
TOOL_GUIDANCE_LARGE = (
"You have access to tools - use them proactively when you need current information or to perform actions. "
"After receiving tool results, use the data to FULFILL THE USER'S ORIGINAL REQUEST. "
"Do NOT describe the structure of tool responses - extract the relevant information and present it conversationally. "
"Tool results are raw data for you to interpret and use, not content to describe or explain. "
"CRITICAL fidelity rule: when you answer a question using a tool result, every specific fact in your "
"reply (names, dates, cast, authors, places, numbers, plot details, product specs) must come from the "
"tool result itself or from the user's own messages. Do NOT supplement tool results with cast, plot, "
"release years, authors, or other specifics from your prior — even if they feel plausible. If the tool "
"returned only a short summary, answer using only that summary; do not extend it with invented detail. "
"If the tool result doesn't contain what the user asked for, say so and offer to look up more rather "
"than filling the gap from memory. "
"When a webSearch result includes a '**Content from top result:**' section, quote its specific facts "
"(names, dates, roles, plot) rather than deferring to the '**Other search results:**' link list. "
"The links are provenance, not a substitute for an answer."
)
# Large models also confabulate on named entities — e.g. gpt-oss:20b produces a
# confident but wrong cast list for the film "Possessor" without calling
# webSearch. The anti-confabulation rule is therefore not a small-model-only
# concern. We keep a shorter version here (large models follow concise
# instructions reliably; repetition and worked examples are only needed for
# small models).
#
# NB: constraints are intentionally phrased without any language-specific
# negative examples ("would you like me to", "if you'd like", etc.) because
# this assistant supports an arbitrary set of languages. We describe the
# BEHAVIOUR to avoid, not English tokens that happen to express it.
TOOL_CONSTRAINTS_LARGE = (
"ACTION REQUESTS — NEVER REFUSE BEFORE CHECKING:\n"
"When the user asks for an action, scan your available tools and call the one whose "
"description covers that action. Do NOT apologise or claim you cannot do it. If "
"nothing in your current list fits, call `toolSearchTool` with a short description "
"of the action before giving up. A false refusal when a matching tool exists is the "
"worst possible reply.\n\n"
"UNKNOWN NAMED ENTITIES:\n"
"When the user asks about a specific named thing (a film, book, song, game, "
"product, person, company, place, event), call webSearch before answering unless "
"you can state concrete, verifiable facts about that exact entity with high confidence. "
"Do NOT confabulate cast, plot, release year, authors, or other specifics from a "
"plausible-sounding prior — if you are not certain, look it up. "
"A diary or memory entry mentioning the entity's name only confirms the topic came "
"up before; it does NOT give you facts you can restate. "
"Do not announce the search or ask permission — just call the tool, then answer. "
"Any phrasing that requests information about a named entity (\"tell me about X\", "
"\"have you heard of X\", and equivalents in any language) is a search trigger, "
"not a capability question about yourself.\n\n"
"ARGUMENTS THE TOOL CAN AUTO-DERIVE:\n"
"Tools may state in their description that an argument has a sensible default "
"(for example getWeather uses the user's current location when none is given). "
"Call the tool in the SAME turn with whatever arguments you have — even zero — "
"and let it fill the rest. Do NOT reply with a clarifying question like \"which "
"location?\" for an argument the tool auto-derives. Concretely: \"how's the "
"weather today\" must trigger getWeather immediately with no arguments, not a "
"question back to the user.\n\n"
"SELF-CONTAINED TOOL ARGUMENTS:\n"
"When you call any tool with a free-form text argument (search queries, lookup "
"strings, question fields — whatever the tool calls them), the string you pass "
"must be a self-contained version of the user's intent. Resolve pronouns, "
"ellipsis, and implicit references from the conversation so far — the tool does "
"NOT see prior turns. If turn 1 was about Harry Styles and turn 2 asks \"what "
"are his most famous songs?\", the argument must name Harry Styles explicitly, "
"not echo the literal utterance. Prefer a compact keyword phrasing over a "
"conversational sentence: \"Harry Styles most famous songs\" beats \"what are "
"his most famous songs\". This applies to every tool you call, not just "
"webSearch."
)
# =============================================================================
# Small Model Prompts
# =============================================================================
TOOL_INCENTIVES_SMALL = (
"Use tools when they can provide better, more accurate responses. "
"Follow each tool's description to decide when to use it. "
"For current information, real-time data, or external lookups - use tools confidently. "
"For greetings and small talk - respond directly without tools."
)
TOOL_GUIDANCE_SMALL = (
"You have access to tools - use them when the task requires external data or actions. "
"After receiving tool results, use the data to answer the user's question conversationally. "
"Extract relevant information and present it naturally - never output raw JSON or data structures. "
"Tool results are YOUR OWN DATA to use when answering — they are not a new message from the user "
"and they are not a prompt for you to interpret. The user's question is in their earlier message "
"above the tool result; the tool result is the material you use to answer it. Do NOT reply by "
"describing the tool result back (\"the text is a collection of search results\", \"you have not "
"asked a specific question\", \"the provided text does not contain a direct question\") — the user "
"already asked their question and the tool already answered it, you just need to state the answer. "
"If the tool result contains facts that address the user's earlier question, synthesise those facts "
"into a direct answer. If it does not, say so briefly and offer to look further — never pretend no "
"question was asked. "
"CRITICAL fidelity rule: when answering using a tool result, every specific fact in your reply "
"(names, dates, cast, authors, places, plot details, numbers) must come from the tool result or "
"from the user's own messages. Do NOT add cast, plot, release years, authors, or other specifics "
"from your prior knowledge — even if they feel plausible. If the tool returned only a short summary, "
"answer using only that summary. If the result doesn't contain what the user asked, say so rather "
"than filling the gap from memory. "
"When a tool result contains a section labelled '**Content from top result:**', pull the specific "
"facts (names, dates, roles, plot, numbers) from that section and state them in your reply. Do NOT "
"defer to the '**Other search results:**' link list by saying things like 'here are some links' or "
"'sources like Wikipedia' — those links are for your reference only; the user wants the facts, not "
"the URLs. If the Content section has the answer, give it; only fall back to mentioning sources when "
"the Content section is empty or clearly off-topic."
)
# Explicit constraints for small models - focused specifically on the greeting case
# without being overly restrictive on legitimate tool use.
# NOTE: Repeated twice (x2) intentionally. Research shows repeating key instructions
# improves instruction-following in smaller models.
# See: "The Power of Noise: Redefining Retrieval for RAG Systems" (arXiv:2401.14887)
# and "Lost in the Middle: How Language Models Use Long Contexts" (arXiv:2307.03172)
# Repetition places the constraint both early (primacy) and late (recency) in the prompt.
# NB: these constraints are intentionally phrased WITHOUT language-specific
# examples of forbidden phrasing ("would you like me to", "I can search", etc.)
# because this assistant supports an arbitrary set of languages. We describe
# the BEHAVIOURS to avoid, not English tokens that happen to express them.
# Small models still get enough structure to follow because each rule is
# stated in imperative form with a concrete trigger + action.
_TOOL_CONSTRAINTS_BASE = """ACTION REQUESTS — NEVER REFUSE BEFORE CHECKING:
When the user asks for an action (open something, navigate somewhere, send a message, look something up, play something, fetch data), scan your available tools FIRST and call the one whose description covers that action. Do NOT apologise, do NOT say "I cannot do that", do NOT describe your limitations — just call the tool. If nothing in your current tool list obviously fits, call `toolSearchTool` with a short description of the action before giving up. A false refusal when a tool exists is the worst possible reply; calling a tool that turns out not to help is recoverable. Treat "I cannot" as a last resort reserved for when both your tool list AND `toolSearchTool` have been exhausted.
GREETING HANDLING:
When the user's message is a greeting or casual social phrase (whatever language), respond directly and warmly WITHOUT calling any tools. Greetings do not require external data.
USER INSTRUCTIONS:
When the user gives you instructions about how to behave or respond (units, brevity, language, tone), acknowledge and respond directly WITHOUT calling tools. These are behavioural instructions, not data requests.
UNKNOWN NAMED ENTITIES:
If the user asks about a specific named thing (a film, book, song, game, product, person, company, place, event) and you do not have concrete factual information about that exact entity, call webSearch in the SAME turn — silently. Do not offer to search, do not ask permission to search, do not announce the search, do not say you have no information and stop. If the query names the entity clearly enough to search, SEARCH — do not ask the user to disambiguate first. Clarifying BEFORE a tool call is a deflection; clarifying AFTER the tool returns nothing useful is fine.
Any phrasing that requests information about a named entity is a search trigger — the request doesn't have to contain the word "search". Treat "tell me about X", "tell me more about X", "what do you know about X", "what can you tell me about X", "have you heard of X", and their equivalents in any language as information requests about X, not as capability questions about yourself. The correct response is to look X up and answer — not to describe what you can or cannot do.
Only skip the lookup if you can state concrete facts about the exact entity (title, year, creator, plot) without guessing. A diary or memory mention of the entity's name only confirms the topic came up — it does NOT give you facts you can state. Never invent plot, cast, release year, themes, or other specifics from prior knowledge. If you do not have facts from a tool result in this turn, you must call webSearch.
ARGUMENTS THE TOOL CAN AUTO-DERIVE:
If a tool's description says it has a default for some argument (for example getWeather uses the user's current location when none is given), call the tool in the SAME turn with whatever arguments you do have — even zero — and let the tool fill the rest. Do NOT ask the user to supply that argument. Do NOT reply with a clarifying question like "which location?" or "where are you?" when the tool's description already states it auto-derives that argument. Concretely: a message like "how's the weather today" must trigger getWeather immediately with no arguments, NOT a question back to the user. Asking for an argument the tool auto-derives wastes a turn and frustrates the user.
SELF-CONTAINED TOOL ARGUMENTS:
Whenever you call any tool with a free-form text argument (a search query, lookup string, question field — whatever the tool names it), the string you pass MUST be a self-contained restatement of the user's intent. Resolve pronouns, ellipsis, and implicit references from earlier turns yourself — the tool does NOT see the conversation history, it only sees the argument you pass. If the previous turn was about "Harry Styles" and the user now asks "what are his most famous songs?", the argument must be something like "Harry Styles most famous songs", NOT "what are his most famous songs". Prefer a compact keyword phrase over a conversational sentence. Never pass the user's literal utterance through when it contains unresolved pronouns, "that", "those", "it", "his", "her", "their", or similar references. This applies to every tool — webSearch, Wikipedia, MCP tools, all of them."""
# Repeat the constraints twice for better instruction-following in small models
TOOL_CONSTRAINTS_SMALL = _TOOL_CONSTRAINTS_BASE + "\n\n" + _TOOL_CONSTRAINTS_BASE
# =============================================================================
# Prompt Assembly
# =============================================================================
def get_system_prompts(model_size: ModelSize) -> PromptComponents:
"""
Get prompt components appropriate for the given model size.
Args:
model_size: The detected model size
Returns:
PromptComponents with all necessary prompt strings
"""
if model_size == ModelSize.SMALL:
return PromptComponents(
asr_note=ASR_NOTE,
inference_guidance=INFERENCE_GUIDANCE,
tool_incentives=TOOL_INCENTIVES_SMALL,
voice_style=VOICE_STYLE,
tool_guidance=TOOL_GUIDANCE_SMALL,
tool_constraints=TOOL_CONSTRAINTS_SMALL,
)
else:
return PromptComponents(
asr_note=ASR_NOTE,
inference_guidance=INFERENCE_GUIDANCE,
tool_incentives=TOOL_INCENTIVES_LARGE,
voice_style=VOICE_STYLE,
tool_guidance=TOOL_GUIDANCE_LARGE,
tool_constraints=TOOL_CONSTRAINTS_LARGE,
)

115
src/jarvis/reply/prompts/prompts.spec.md Normal file
View File

@@ -0,0 +1,115 @@
## Prompts Module Spec
This module provides model-size-aware prompt generation for the reply engine.
### Problem Statement
Small models (1b, 3b, 7b parameters) lack the reasoning capacity to infer when NOT to use tools. When given prompts like "Proactively use available tools," they may incorrectly call tools for simple greetings like "hello" or "ni hao" because they cannot distinguish between:
- Requests that require tools (weather, search, data retrieval)
- Simple conversation (greetings, small talk, general knowledge)
### Solution: Model-Size-Aware Prompts
The module detects model size from the model name and selects appropriate prompts:
| Model Size | Detection Pattern | Tool Prompts |
|------------|-------------------|--------------|
| SMALL | `:1b`, `:3b`, `:7b`, `gemma4` | Conservative — explicit "DO NOT use tools for greetings" + worked negative examples + repetition |
| LARGE | All others (8b+) | Proactive — "use tools confidently" + short anti-confabulation + auto-derive clause |
### Architecture
```
src/jarvis/reply/prompts/
├── __init__.py # Public exports
├── system.py # Base constants (ASR_NOTE, VOICE_STYLE, etc.)
├── model_variants.py # Model detection + size-specific prompts
└── prompts.spec.md # This file
```
### Public API
```python
from jarvis.reply.prompts import (
ModelSize, # Enum: SMALL, LARGE
detect_model_size, # (model_name: str) -> ModelSize
get_system_prompts, # (model_size: ModelSize) -> PromptComponents
PromptComponents, # Dataclass with all prompt strings
)
```
### Prompt Components
Both model sizes share these base components:
- `asr_note`: Voice transcription error handling
- `inference_guidance`: Prefer inference over clarification
- `voice_style`: Concise, conversational responses
Model-size-specific components:
- `tool_incentives`: When/how aggressively to use tools
- `tool_guidance`: How to handle tool results (both sizes get the anti-confabulation fidelity rule and the "quote Content from top result, don't deflect to links" rule)
- `tool_constraints`: Explicit behaviour rules. Present on BOTH sizes — the
large variant is a shorter restatement of the named-entity and tool-
auto-derive rules because gpt-oss:20b and similar also confabulate
specifics for unfamiliar entities and occasionally ask for arguments
(e.g. `location` for `getWeather`) the tool already auto-derives.
### Small Model Tool Constraints
Small models receive **focused constraints** that are **repeated twice (x2)** in the prompt.
The constraints target specific cases where small models incorrectly call tools, without restricting
legitimate tool use (like web search for current information).
This leverages research findings on prompt repetition:
- **"Lost in the Middle: How Language Models Use Long Contexts"** (arXiv:2307.03172)
Shows models attend more to text at the beginning (primacy) and end (recency) of prompts.
- **"The Power of Noise: Redefining Retrieval for RAG Systems"** (arXiv:2401.14887)
Demonstrates that repeating key instructions improves instruction-following.
Sections (both sizes — small repeats twice):
- **GREETING HANDLING** — greetings / social phrases in any language must not trigger tools.
- **USER INSTRUCTIONS** — behavioural instructions (units, brevity, language, tone) are acknowledged directly.
- **UNKNOWN NAMED ENTITIES** — any information request about a specific named entity calls webSearch in the SAME turn, silently; the enumeration of request phrasings ("tell me about X", "have you heard of X", etc. — in any language) is framed as a semantic category, not as blacklisted English tokens.
- **ARGUMENTS THE TOOL CAN AUTO-DERIVE** — if a tool's description says it has a default for an argument (e.g. getWeather → user's location), call the tool without asking the user for that argument.
- **SELF-CONTAINED TOOL ARGUMENTS** — free-form text arguments passed to any tool (search queries, lookup strings, etc.) must be a self-contained restatement of intent with pronouns and ellipsis resolved from conversation history. Tools don't see prior turns. This applies to every tool, including MCP tools we don't own — the rule lives in the system prompt rather than each tool's schema so it generalises.
**Design Rationale:**
- Constraints are narrowly scoped to specific problematic cases
- Covers greetings AND behavioral instructions (both don't require tools)
- Includes a positive rule for unknown named entities — small models otherwise deflect ("I don't have information about X") instead of calling webSearch
- It does NOT restrict web search for current information queries
- It does NOT prevent tools from being used for legitimate tasks
- Small models should still use tools when the user asks about news, weather, etc.
### Integration with Reply Engine
The reply engine detects model size early and passes it to `_build_initial_system_message()`:
```python
from jarvis.reply.prompts import detect_model_size, get_system_prompts
model_size = detect_model_size(cfg.ollama_chat_model)
prompts = get_system_prompts(model_size)
# Build system message from prompts.to_list()
```
### Language Agnosticism
All prompts are language-agnostic:
- Greetings list includes examples in multiple languages
- No English-specific patterns or assumptions
- Intent detection based on conversation type, not specific words
### Testing
1. **Unit tests** (`tests/test_prompts.py`):
- Model size detection for various model names
- Prompt component selection
2. **Eval tests** (`evals/test_greeting_no_tools.py`):
- Greetings in multiple languages don't trigger tools
- Tool-requiring queries still trigger tools

66
src/jarvis/reply/prompts/system.py Normal file
View File

@@ -0,0 +1,66 @@
"""
Base system prompt constants shared across all model sizes.
These prompts are language-agnostic and focus on core assistant behavior.
"""
from dataclasses import dataclass
from typing import Optional
# Voice/ASR clarification - accounts for transcription noise
ASR_NOTE = (
"Input is voice transcription that may include: errors, missing words, filler words (um, uh, like), "
"or unrelated speech captured before the user addressed you. "
"Extract the user's actual request/question directed at you - ignore any preceding chatter or conversation fragments. "
"Prioritize their intent over literal wording."
)
# General inference guidance - prefer action over clarification
INFERENCE_GUIDANCE = (
"Prioritize reasonable inference from available context, memory, and patterns over asking for clarification. "
"When you make assumptions or inferences, be transparent about them. "
"Only ask clarifying questions when the request is genuinely ambiguous and inference would likely be wrong."
)
# Voice assistant communication style - concise, conversational
VOICE_STYLE = (
"Keep responses concise and conversational since this is a voice assistant. "
"Two to three sentences maximum. Prioritize clarity and brevity - users are listening, not reading. "
"Avoid unnecessary elaboration unless specifically requested. "
"Do NOT offer follow-up suggestions or ask if the user wants more info - just respond directly. "
"IMPORTANT: Always respond in natural language - never output JSON, code, or structured data as your response. "
"NEVER use markdown formatting in your replies: no asterisks for emphasis (**bold**, *italic*), "
"no hashes for headings, no bullet points or numbered lists, no backticks. "
"The text you produce is spoken aloud by a TTS engine that reads these characters literally — "
"asterisks are read as 'asterisk asterisk'. Write plain sentences only."
)
@dataclass
class PromptComponents:
"""
Collection of all prompt components for a specific model size.
All components are combined in _build_initial_system_message() to form
the complete system message.
"""
asr_note: str
inference_guidance: str
tool_incentives: str
voice_style: str
tool_guidance: str
tool_constraints: Optional[str] = None # Only for small models
def to_list(self) -> list[str]:
"""Convert to list of non-empty prompt strings."""
components = [
self.asr_note,
self.inference_guidance,
self.tool_incentives,
self.voice_style,
self.tool_guidance,
]
if self.tool_constraints:
components.append(self.tool_constraints)
return [c for c in components if c]