Transform isair/jarvis into a Discord-controlled voice assistant running on the Ubuntu VNC desktop, keeping the mature ~39k-line Python brain intact. - bot/ (Node + bun, discord.js): /자비스 slash commands (ephemeral), voice channel join + voice receive/playback, pluggable VNC screen broadcast (selfbot live / noVNC / screenshot) - bridge/ (Python, Flask): wraps jarvis STT + run_reply_engine + Piper TTS behind a thin localhost HTTP API - .env.example, scripts/ (start_bridge/start_bot/dev), README rewrite, docs/language-comparison.md and docs/vnc-xfce-setup.md Language decision: hybrid (Python brain + Node/bun Discord layer) because Discord blocks bot video; native screen broadcast only works via a Node selfbot library.
12 KiB
Deprecated: The evaluator is no longer called from the reply engine. The task-list planner (
planner.spec.md) replaces its per-turn correction role. This file is preserved for reference only.
Agentic-Loop Evaluator Spec
Purpose
After each agentic-loop turn that produces natural-language content (as opposed to a tool call), a lightweight LLM decides whether the loop should terminate (the agent has done what it can) or continue (a tool in the agent's allow-list could directly perform the user's expressed action but the agent replied in prose instead).
The axis is deliberately binary: from the agentic loop's perspective, "satisfied" and "needs_user_input" are the same terminal state — both mean stop looping and hand back to the user. Collapsing them removes the accidental third class that the previous contract had, where a coherent-but-wrong prose reply (agent describes what it could do, but doesn't do it) was being marked satisfied and shipped.
Input contract
evaluate_turn(user_query, assistant_response_summary, available_tools, turns_used, cfg, invoked_tools=None):
user_query(str): the redacted user query that opened this reply. Defensively re-redacted on entry.assistant_response_summary(str): the natural-language content produced by the chat model on the current turn. Redacted on entry in case the model echoed sensitive user text.available_tools(list of(name, one_line_description)or(name, one_line_description, input_schema)tuples): the agent's current allow-list. Engine-supplied, not user data, so not redacted. When theinput_schemaslot is populated (JSON Schema dict withpropertiesand optionalrequired), the evaluator prompt renders each tool astoolName(param: type required, ...): descriptionso the judge emitstool_call.argumentswith exact parameter names. Without the schema, small evaluator models hallucinate plausible-looking argument keys (queryinstead ofsearch_query) that pass the engine's allow-list check but fail the tool's own validation, producing a loop of validation-error tool results.turns_used(int): number of loop turns consumed so far.cfg: config object providing the base URL, model, and timeout.invoked_tools(optional list of(name, args_summary, result_summary)tuples): tools that have ALREADY executed during this reply, including direct-exec and model-emitted calls. Lets the evaluator distinguish "agent hasn't tried the tool" (→ nudge) from "tool already ran successfully, the chat model just failed to narrate the result" (→ terminal, do not re-invoke). Without this context, a small chat model that replies in prose after a successful direct-exec causes the evaluator to keep re-requesting the same tool indefinitely. Results are redacted defensively because tool output can echo user-provided text.
Output contract
EvaluatorResult(terminal: bool, nudge: str = "", reason: str = "", tool_call: Optional[dict] = None).
terminal:Truemeans exit the loop and deliver the reply;Falsemeans keep looping.nudge: whenterminal=False, a short directive to the agent telling it which tool to use and what to do with it. Injected into the next turn's system message as[Agent nudge: ...], lasts exactly one turn. Empty whenterminal=True.reason: free-text log hint only. Never shown to the user.tool_call: optional structured{"name": str, "arguments": dict}intent. When the judge has identified both a specific tool (that appears in the toolbox) and its arguments — either by salvaging a garbled tool-call attempt or by spotting an obvious missed invocation — it populates this field in addition to the free-formnudge. The engine uses the structured form to execute the tool directly on behalf of the agent, bypassing small chat models that ignore textual nudges.Nonewhen the judge is nudging for prose, is uncertain about arguments, or is returning terminal. The engine rejects the call ifnameis not in the current allow-list, falling back to the text-nudge path.
Rubric
Return continue (non-terminal) when ALL of the following hold:
- the user expressed a clear action or request, AND
- a tool in the agent's toolbox could directly perform it, AND
- the agent's turn was prose (an offer, a suggestion, a description of what it could do) instead of invoking that tool.
Return terminal when the agent genuinely finished: delivered a real answer, successfully completed the action, or truthfully said it cannot do this because no tool fits.
Return continue when the agent's turn is garbled — raw tool-protocol markers (tool_code / tool_output blocks), special sentinel tokens (<unused88> and other <unused…> variants), bare tool_calls: text, truncated JSON, or code/data dumps where a prose answer should be. The deterministic _is_malformed_model_output guard in the engine catches the known shapes before the evaluator even runs; the evaluator's garbled-turn clause is defence-in-depth for novel leaks the guard has not learned yet.
When the garbled turn encodes a failed tool-call attempt (e.g. a tool_code block wrapping google_search.search(query="…"), a bare tool_calls: [{"name": "webSearch", "arguments": {…}}] JSON blob, or a <unused…> block wrapping a tool invocation), the evaluator salvages the intent: extract the intended tool and arguments from the garbled text, validate that the tool name appears in the turn's allow-list, and name the tool + args both in the free-form nudge and in the structured tool_call field, e.g. nudge="call webSearch with query='sam smith biography'", tool_call={"name": "webSearch", "arguments": {"search_query": "sam smith biography"}}. The engine prefers the structured form: when tool_call is present and the name is in the allow-list, the engine runs the tool directly on behalf of the agent via the normal run_tool_with_retries path (same allow-list check, schema validation, and redaction guards as a model-emitted call). The structured path exists because small chat models routinely see the textual nudge and reply with more prose instead of actually emitting the tool-call protocol — one or two nudges burned, nudge cap fires, user gets an ungrounded reply. Unrecoverable shapes (truncated JSON with no name, bare <unused88> sentinels, random data dumps) fall back to a "produce a natural-language reply" nudge with tool_call=None. Arguments absent from the garbled turn must not be fabricated — salvage is strictly extraction.
Prompt contract
Strict JSON {"terminal": bool, "nudge": "...", "reason": "...", "tool_call": {"name": "...", "arguments": {...}} | null}, no prose, no code fences. The parser is lenient (strips markdown fences, extracts embedded JSON objects). tool_call is optional and defaults to null; malformed shapes (missing name, non-string name, non-dict arguments) are normalised to null or an empty arguments dict rather than causing a parse failure.
Fail-open behaviour
Any of the following collapse to EvaluatorResult(terminal=True, reason="evaluator_failed_open"):
- Missing base URL or resolvable model.
- Timeout, connection error, or any other exception from the LLM call.
- Empty response from the LLM.
- JSON parse failure.
- Missing or non-boolean
terminalfield.
The fail-open choice was flipped from the previous contract (which defaulted to continue). Biasing toward terminal is safer: spinning in a broken evaluator loop is worse than shipping a possibly-weak reply. agentic_max_turns remains as a hard backstop, and the nudge cap (evaluator_nudge_max) prevents infinite ping-pong even if the evaluator is live but consistently returns continue.
Timeout
Shares llm_digest_timeout_sec (default 8 s) with memory/tool digests.
Model resolution
_resolve_evaluator_model(cfg) picks the first non-empty candidate:
cfg.evaluator_model(explicit override)cfg.intent_judge_model(small, already warm from wake-word path)cfg.ollama_chat_model(last resort)
Gating
cfg.evaluator_enabled:
None(default) — auto: ON for SMALL models, OFF for LARGE. Large models terminate on the first natural-language content.True/False— force on/off regardless of model size.
Relationship to the agentic loop
- Only invoked after a turn produces natural-language content. Tool-call turns bypass the evaluator and keep looping.
- Malformed-JSON fallback replies (canned error text) bypass the evaluator and terminate immediately.
- On
continuethe engine stashes the nudge inpending_nudge; the next turn's system-message rebuild appends[Agent nudge: <text>]at the end of the first system message and clears the slot. So each nudge lasts exactly one turn — if the model keeps producing prose, the evaluator fires again and generates a fresh nudge. - On
continuewith a structuredtool_callwhosenameis in the current allow-list AND is nottoolSearchTool, the engine also stashes it inpending_tool_call. At the top of the next loop iteration — before any chat LLM call — the engine synthesises an assistant message carrying thetool_callspayload, runs the tool viarun_tool_with_retries, records the tool signature inrecent_tool_signaturesfor duplicate suppression, and appends the tool result with the same compound-query remainder hint the model-emitted path uses. The textual nudge is cleared for that turn (the tool has run, no need to also shout the directive at the model). This is the actual recovery path for small models: the evaluator-directed tool execution happens deterministically, the chat model only has to synthesise a reply from the tool result on the following turn. Tool calls that fail the allow-list guard, or that nametoolSearchTool(whose allow-list-widening logic lives only on the model-emitted path), fall through to the textual-nudge path so the safety boundary is never bypassed. - Before direct-execution, the engine validates
argumentsagainst the tool'sinputSchema. An unknown argument key (e.g. evaluator emittedquerywhen the tool requiressearch_query) or a missing required key rejects the call. Rather than consuming a nudge-budget slot (which would punish the chat model for the evaluator's hallucination), the engine enrichespending_nudgewith a concrete schema hint —webSearch(search_query: string required)— and hands control back to the chat model for this turn. The chat model sees both the schema hint and its original[Agent nudge: ...]block and is expected to emit a propertool_callspayload itself. Type-checking is intentionally not enforced here; tool implementations own that, and pre-checking types would reject too many borderline cases. - Before stashing
pending_tool_call, the engine checks whether(name, arguments)duplicates a recent signature inrecent_tool_signatures. Argument keys are lower-cased for the comparison so evaluator case-flips (urlvsURL) collide. On a hit the loop terminates with the latest plausible candidate reply instead of re-executing. This is defence-in-depth: the primary mechanism preventing duplicate execution is theinvoked_toolscontext fed to the evaluator itself (so the judge declines to re-request a tool that has already run); the guard catches the residual case where a small evaluator ignores that context. cfg.evaluator_nudge_max(default 2) caps how many textual nudges can be issued per reply. Direct-executabletool_callresults do NOT consume the nudge budget — they are deterministic actions, not directives the model can ignore. A structuredtool_callthat falls back to the textual-nudge path (allow-list miss, ortoolSearchTool) DOES count. Once the cap is reached, the next textual-nudgecontinueis overridden to terminal. This stops nudge ping-pong when the model consistently ignores the directive.- The loop tracks the latest plausible candidate and delivers it when
agentic_max_turnsis hit.
Tests
tests/test_evaluator.pycovers parse edge cases, terminal and continue-with-nudge paths, timeout / connection-error fail-open (now terminal), missing-config fail-open, redaction, and the available-tools payload shape.tests/test_engine_tool_search_loop.pycovers the integration with the agentic loop including the continue-then-nudge-then-tool-call sequence.